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ABSTRACT 


The  ability  of  computer  users  to  specify  problem  solu¬ 
tions  with  the  help  of  example  solutions  was  investigated  as  a 
function  of  the  user's  background  and  experience,  various  levels 
of  processing,  and  various  levels  of  problem  complexity. 

Two  participant  groups  (programmers  and  bookkeepers/ 
accountants)  working  with  three  levels  of  problem  complexity 
and  three  levels  of  processor  complexity  were  used.  The  ex¬ 
periment  task  employed  in  this  study  required  specification  of 
a  logic  for  selection  of  a  Navy  task  force. 

The  results  showed  that  specification  of  problem  solutions 
by  example-solutions  led  to  low  rates  of  errors-of-commission. 
Further,  the  rate  of  errors-of-omission  was  significantly 
affected  by  the  degree  of  generalization  of  the  example  inputs 
by  the  automatic  processor.  The  results  also  suggested  that  the 
strategy  used  in  developing  the  example  solutions  may  be  a 
significant  factor  in  the  generation  of  accurate  problem  solution 
specifications . 
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INTRODUCTION 


The  processes  by  which  computer  programs  are  de¬ 
veloped  and  tested,  as  well  as  the  process  of  revising  estabV'shed 
programs,  are  apparently  so  incompatible  with  the  human 
memory  and  information  processing  capabilities  that  computer 
programs  often  cannot  be  synthesized  in  a  precise  manner. 
Instead,  they  are  often  developed  in  a  "cut  and  try"  manner,  w'^ere 
the  development  proceeds  on  what  is  essentially  an  iterative 
refinement  basis.  For  instance,  Ramanoorthy  (1978) 
proposed  a  design  methodology,  shown  in  Figure  1  ,  that 
explicitly  recommends  repeated  revisions  of  the  four  elements 
of  the  design:  1.  requirement  and  specification;  2, 
design;  3.  implementation;  4.  evaluation  nd  validation. 

A  model  that  explains  the  success  of  a  "cut  and  try"  design 
method  also  points  out  its  major  weakness:  If  program  errors 
are  eliminated  one-by-one  as  testing  proceeds,  the  program 
can  only  be  improved  providing  new  errors  are  not  introduced 
when  old  errors  are  corrected.  Even  moderately-sized 
computer  programs  '‘'Ten  require  r<  >etitivc  debugging  and 
patchup  to  make  them  acceptable;  but  errors  in  large 
programs  may  not  be  discovered  until  the  program  has 
actually  been  used  in  the  field.  Further,  the  program 
development  process  is  uncertain  since  program  development 
schedules  and  costs  freqe.  ’y  exceed  estimates  thought  to 
be  more  than  adequate. 

The  research  reported  here  involved  an  investiga¬ 
tion  of  the  ability  of  computer  users  to  specify  problem 
solutions.  This  ability  ;s  evalua.od  as  '  function  of  the 
user's  background  an'1  experience,  the  complexity  of  the 
problem  to  be  solved,  the  complexity  of  the  avail  a '■  'e 
processor  (computer),  and  the  available  feedback  aids. 

In  order  to  provide  background  information,  a  review 
of  current  developments  in  software  engineering  is  presented. 

This  rev;ew  indicates  that  developments  in  structured  programming 
are,  to  some  extent,  based  upon  limitations  in  human  information 
processing  capabilities;  but  are  mainly  efforts  to  enhance 
computer  program  quality,  an  issue  which  is  conceptually 
different  from  the  specification  of  problem  solutions.  Further, 
developments  in  automatic  programming  technology,  by  which 
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computer  programs  can  be  automatically  synthesized,  have  indicated 
that  ultimately  the  user  may  be  required  only  to  specify  accurately 
the  problem  solution, since  the  computer  programs  themselves 
will  be  automatically  synthesized  from  solution  specifications. 

Developments  in  Software  Engineering 

Several  possible  reasons  exist  for  the  difficulties 
involved  in  the  design  and  implementation  of  computer  programs. 

For  instance,  it  may  be  that  a  given  task  requires  attention  to  so 
much  detailed  information  and  specialized  knowledge  of  machine 
information  (often  unique  to  a  particular  computer  installation) 
that  the  human  analyst  and/or  programmer  might  have  extreme 
difficulty  in  remembering  and  recalling  the  required  information. 
Another  po  ,  .ible  reason  may  be  that  the  step-by-step  nature 
of  the  programming  task  prevents  adequate  consideration  of  the 
overall  computer  program.  WK?‘  ver  the  true  reason,  a 
considerable  number  of  ongoing  efforts  are  designed  to  address 
such  problems.  These  efforts  fall  into  the  following,  somewhat 
overlapping,  categories: 

1  .  Design  methodology, 

2.  Programming  languages, 

3.  Automatic  programming, 

4.  Program  test  a-  validation. 

Design  Method  fogy 

Structured  software  design  methodologies  have  been  and 
continue  to  be  developed  based  on  the  belief  that  the  process  of 
producing  a  computer  program  governs  the  quality  of  the  program. 
As  described  by  Bergland  (1979),  these  structured  techniques 
impact  on  all  components  of  the  software  production  process. 

Various  princip’  _?s  of  structured  design  have  been 
discussed  by  Yourdon  &  Constantine  (1979),  and  by  Jensen  & 

Tonies  (1979).  Yourdon  S  Constantine  have  indicated  that  the 
limitations  of  human  information  processing  should  be  used  as 
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a  basis  for  defining  a  computer  program  design  process.  Further, 
they  suggest  that  Miller's  "Magical  Number  7  ±  2"  item  limita.ion 
on  human  short-term  memory  capability  (Miller,  1956)  should  be 
a  guiding  principle  in  program  design.  Yourdon  and  Constantine 
have  identified  program  complexity  in  "human  terms"  as  being 
related  to  program  size,  and  have  suggested  that  there  may  be 
an  effective  limit  to  the  number  of  pieces  (modules)  into 
which  a  problem  can  be  decomposed. 

In  a  somewhat  similar  approach,  Jensen  and  Tonies  (1979) 
recognized  that  characteristics  ol  the  human  pruulem-solving 
process  must  be  considered  while  formulating  a  software 
design  process.  These  authors  have  suggested  that  software 
design  should  be  broken  into  chunks  which  are  more  amenable 
to  human  comprehension.  This  is  genor^’ly  accomplished 
by  attacking  the  problem  at  an  abstract  ’  vol  and  then  proceeding 
to  more  detailed  levels  of  resign. 

Although  Yourdon  and  Constantino  have  provided  what  is 
considered  to  be  an  initial  human  processing  capability  basis 
for  software  development,  short— term  memory  is  by  no 
means  the  only  human  information  processing  capability 
involved  in  software  development.  Additional  factors  of  interest 
include  the  effect  of  problem  and  machine  complexity,  and  the 
utilization  of  abstract  nru  concrete  levels  of  communication  in  each 
step  of  the  design  proce  . 

Programming  Languages 

The  continued  development  of  new  programming  languages 
is  based  upon  a  rationale  similar  to  that  given  above,  i.e.  ,  that 
the  process  of  generating  a  program,  which  is  influenced  by  the 
programming  language, ultimately  governs  program  quality.  Many 
new  programming  languages  are  a  result  of  previous  work  by  Dahl, 
Dijkstra,  &  Hoare  (1972)  ami  by  Dijkstra  (1976).  For  example,  the 
PASCAL  programming  language  (Welsh  &  McKeag,  1980)  and  its 
extensions,  as  well  as  the  newer  ADA  language  (Wegner,  1980)  were 
both  developed  from  this  earlier  ,,ork.  The_>u  approaches,  which  involve 
the  structured  step-by-step  refinement  of  information,  have  con¬ 
siderable  appeal  from  a  mathematical  viewpoint  and  apparently  en¬ 
hance  the  development  of  "correct"  machine  code;  however,  as  men¬ 
tioned  above,  the  present  investigation  is  not  designed  to  address  the 
development  of  "correct"  machine  code,  but  rather  to  investigate 


4 


factors  that  affect  the  rapid  and  accurate  specification  of  problem 
solutions  from  which  computer  programs  are  generated. 

Automatic  Programming 

Another  approach,  which  falls  into  both  the  automatic 
programming  category  and  the  program  language  category  listed 
above,  combines  results  from  both  programming  language  and 
artificial  intelligence  research.  This  re1  arch  has  already 
suggested  a  dramatic  reduct  n  in  the  effort  required  to  provide 
machine  code  and  has  highlighted  the  increasing  irrv  stance  of 
adequate  problem  analysis  and  correct  pro. -ram  specirication. 

It  is  generally  agreed  by  researchers  in  Drogramming  languages 
and  in  artificial  intelligence  that  the  process  of  generating  a 
computer  program  can  be  logically  divided  into  two  steps: 
"program  specification"  and  "program  synthesis"  (Biermann, 
1976).  Other  researchers  have  referred  to  these  steps  by 
other  names:  "top  part,  bottom  part"  (Prywes,  1977), 

"program  acquisition,  program  coding"  (McCune,  "977),  and 
"formulating  the  plan,  formalising  the  plan"  (Wile,  Ba’.ceT  & 
Goldmann,  1977).  It  is  generally  agreed  that  once  a  program 
has  been  specified,  program  coding  can  proceed  without  dif¬ 
ficulty.  Various  program  synthesis  (coding)  systems  have  been 
constructed  and  demonstrated  by  the  researchers  cited  above, 
among  others . 

The  first  of  the  two  steps,  program  specification,  has 
been  studied  from  various  viewpoints,  including  artificial 
inte’.ligenc  ,  as  reported  ;n  a  survey  by  Biermann  (1976), 
where  inputs  from  the  user  might  include  examples,  or  input- 
output  specifications,  or  natural  language  commands;  and 
the  position  of  Wile,  et  al.  (1977)  who  insist  that  the  user  is 
respon,  ;Klc  for  formulating  the  plan.  At  present,  p*'  gram 
specification  is  the  mn1  '  impediment  to  automatic  pm .  ramming; 
yet  research  to  d’Be  ha^.  generally  concentrated  on  the  machine 
side  of  the  man-machine  interface,  in  which  the  human 
programmer  (user)  is  considered  as  merely  an  inputter 
of  program  specifications.  But  since  the  user  is  the 
key  element  in  the  automatic  programming  process, 
machines  should  be  adapted  to  human  capabilities  >  ithqr  than 
humans  to  machine  processes.  Major  issues  here  in¬ 
volve  the  unique  capabilities  of  the  human  user  in  specifying 
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program.  ,  welL  as  the  influence  ui  machine  processing 
(generalizations)  and  feedback  aids  on  the  specification  process. 

Program  Test  and  Validation 

An  enormous  number  of  methods  and  tools  for  testing, 
inspecting,  and  checking  computer  programs  have  been  devised 
(Meyers,  1979).  Such  tools,  however,  are  frequently  not 
geared  to  testing  programs  as  problem  solutions,  i.e.,  does 
the  program  solve  the  problem  at  hand? 

Two  requirements  govern  program  test  and  evaluation 
efforts.  First,  the  testing  must  in  pari  be  problem  independent 
(e.g.,  attempts  to  determine  whether  a  program  will  reach 
completion);  second,  part  of  the  testing  must  be  problem 
dependent  (e.g.,  to  determine  whether  a  given  p,  ograr  solves 
the  problem  intended  to  be  solved).  The  first  of  these  re¬ 
quirements  is  a  function  of  the  program  code  only.  ~rMe 
second  is  dependent  in  part  on  the  availability  of  a  correct 
specifu  ation  of  the  problem  solution.  This  is  the  subject  of 
the  researcl  i  reported  here. 

Use  of  Computer  Resources 

Specification,  testing,  and  use  of  computer  programs 
require  interactive  communication  between  the  user  and  the 
computer  system .  The  level  of  abstraction  necessary  to 
specify  a  problem  solution  essentially  defines  the  level  of  com¬ 
munication  at  the  system  interface.  But  whatever  the  aids  are 
that  prove  to  be  superior  in  supporting  the  desired  level  of 
communication,  they  too  require  '.Me  use  of  computer  resources, 
i.e.,  of  processing  time  and  memory.  Thus,  a  need  exists  to 
determine  the  nature  of  the  tradeoff  between  total  system  per¬ 
formance  (speed  and  accuracy  of  communication)  and  computer 
resources  in  order  to  apply  systems  more  efficiently  to  situa¬ 
tions  in  which  resources  are  limited.  Determination  of  the 
exact  nature  of  that  tradeoff,  in  order  to  enable  a  system 
designer  to  consider  such  issues  along  with  all  other  resource 
demands  when  designing  the  total  system,  is  one  objective  of 
the  present  research  effort. 
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Need  for  Flexible  Communication 


There  is  another  way  to  view  the  user/computer  process 
considered  here.  In  the  design  of  computer  interface  systems,  a 
desipner  typically  views  the  user  with  regard  only  to  his/ner 
level  of  knowledge  about  the  system.  For  instance,  Schneiderman 
(1980),  in  an  extensive  review  of  interface  design  methods,  has 
identified  various  design  check  lists  (!-iansen  (1971),  Wasserman 
(1973),  Pew  &  Rollins  (1975),  Gaines  &  Fa.ey  (1975),  Cheriton 
(1976),  Gebhardt  &  Stellmacher  (1978),  Turoff,  Whitescarver 
&  Hiltz  (1978),  and  Kennedy  (1974))  in  which  many  worthwhile 
suggestions  are  offered,  yet  the  cardinal  rule  agreed  upon  is 
"know  the  user." 

Such  limited  views  of  user  populations  permit  scaling 
of  input  only  according  to  user  knowledge  of  the  system.  "!"hus, 
the  beginning  users  are  typically  presented  with  a  "menu,” 
list  mg  a  set  of  typical  oput  sec.  ices  tor  potential  use.  The 
experienced  user,  however,  is  typically  provided  wit;i  a  set  of 
general  commands,  from  w^ich  he  can,  v\  th  great  freedom, 
design  his  own  specific  input  commands. 

This  view,  though  important ,  neglects  another  mie-'face 
dimension  which  is  critical  to  efficient  and  effective  com''-  men¬ 
tion.  In  every  communication,  whether  it  be  between  a  user 
and  a  machine,  or  person  to  person,  there  is  a  gr  .at  amount 
of  information  left  unstated:  information  that  is  assumed  either 
to  be  known  or  to  be  irrelevant.  For  instance,  in  user/computer 
communications ,  much  information  is  stored  in  memory,  system 
status  registers,  I/O  buffers,  etc.,  tnat  u>  typical’y  not  com¬ 
municat'd.  This  uncommunicated  information  is  often  the 
key  to  such  difficult-to-answer  questions  as:  Wi y  didn’t  the 
system  respond  the  way  I  anticipated?  If  !  hit  anot'^  'r  RFT'JRN, 
will  I  lose  the  data  just  entered?  In  what  mode  is  the  computer 
operating?  What  are  alternative  commands?  How  car  >  get  this 
system  to  process  my  data  without  adding  1  to  each  entry?  ,  etc. 

In  general,  the  issue  at  hand  is:  how  can  a  user  establish  com¬ 
munication  or  generate  commands  when  unsure  of  the  correct  way 
to  proceed? 

The  existence  of  such  problems  suggests  that  multi¬ 
level  communication  of  both  abstract  and  concrete  information  (i. 
e.,  both  general  statements  and  examples)  may  provide  a  method 
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for  effective  2-way  communication  between  users  and  computers. 

In  such  a  situation,  if  the  effect  of  a  general  command  is  not 
understood  by  a  user,  the  computer  can  illustrate  by  presenting 
examples  of  specific  c..;es.  Thus  a  user  can  be  provided  with 
general  program  insm  ctions  and  w’th  detailed  solution  examples. 
If  given  general  program  instructions,  the  computer  can  return 
example  solutions.  If  given  detailed  solution  examples,  the  computer 
can  generalize  from  them  and  return  general  instructions  as  well 
as  additional  example  solutions.  Such  multi-level  communications 
are  used  extensively  and  effectively  in  person-to-person  communica¬ 
tions.  High  levels  of  abstraction  provide  eff’ Tent  compact  com¬ 
munications;  examples  (concrete  or  low-level  abstraction)  are  used 
to  enhance  the  understanding  of  what  is  communicated  in  abstract 
terms . 


With  respect  to  the  user/computer  communication  process 
described  above,  we  note,  first,  that  a  user's  capability  to  soecify 
a  program  quickly  and  accurately  (i.e. ,  to  specify  the  solution  to 
his/her  problem)  is  not  only  important,  but  is  r  -mined  as  a 
joint  function  of  the  type  of  problem,  the  available  machine  capability, 
and  the  availability  of  feedback  aids.  Second,  the  machine 
capability  necessary  to  support  the  required  problem  solution  specifica¬ 
tions  must  be  translated  into  computer  resources,  and  the  user's 
performance  capabilities  must  be  established  as  a  function  of  these 
resources.  Finally,  when  multiple  levels  of  information  abstraction 
are  available,  the  capabilities  of  the  users  to  specify  actual  programs 
must  themselves  be  specified. 


METHOD  OF  APPROACH 

Recognizing  that  specification  of  a  problem  solution  is  the 
first  step  in  implementing  the  solution  and  in  time  may  be  the  only 
step  reauired  for  specification  of  a  computer  program,  the  ob¬ 
jectives  of  the  work  reported  here  were  as  follows: 

1  .  To  determine  if  different  levels  of  abstract  in¬ 
formation  enhance  a  user's  :  ,'eed  and  accuracy  in 
specifying  problem  solutions; 

2.  To  determine  the  effects  of  problem  complexity, 
processor  complexity,  and  user  population  ex¬ 
perience  and  background  on  the  speed  and  ac¬ 
curacy  of  specifying  problem  solutions; 


3.  To  determine  the  computer  resources  required 
to  support  high  levels  of  speed  and  accuracy  in 
specifying  problem  solutions. 

Since  planning,  designing,  writing,  and  testing  of  a 
computer  program  are  all  required  to  implement  and  determine 
the  adequacy  of  a  problem  solution,  the  process  of  actually  de¬ 
veloping  n  computer  program  was  used  as  an  initial  mode!  for 
identifying  a  suitable  task  for  the  experiment.  I"  particular, 
literature  describing  various  types  of  programming  errors  was 
examined  in  detail.  Thayer  (1976)  and  Motley  (1977)  for  example, 
have  developed  a  technique  for  coding  programming  errors, 
as  well  as  statistics  for  measuring  frequency  of  occurrence 
of  different  types  of  errors.  Such  efforts  have  shown  that 
programming  errors  classified  as  "logical”  constitute  a  larger 
percentage  of  errors  than  any  other  type.  These  authors  have 
distinguished  logical  errors  from  such  other  categories  of  errors 
as  computational  errors,  input/ output  errors,  data  handling  errors, 
etc.  Their  conclusion,  however,  was  that  by  far  the  greatest 
incidence  of  programming  errors  fell  into  the  logical  category. 

The  term  "logical,"  as  was  used  here,  is  defined  in 
a  somewhat  broader  context  than  that  of  ei  .her  Motley  or 
Thayer.  For  purposes  of  this  effort,  a  "logical  problem"  is 
defined  as  a  problem  wHere  the  solution  is  a  logical  fu-'-tion  of 
known  independent  factors.  Using  this  defini.  on,  errors  may 
arise  if  one  neglects  the  fact  that  a  change  in  a  single  variable 
may  change  the  total  result  (i.e.,  each  independent  variable  may 
be  important  to  the  solution).  For  background,  a  brief  discussion 
of  logical  functions  as  a  basis  for  the  am^oach  presented  is 
provided . 


Loaica’  ^unctions 


N 

If  N  logical  variable^  exist,  it  is  possT’e  to  form  2 
logical  combinations,  and  2-  possible  logical  functions  of  those 
variables  Thus,  if  three  logical  variables  exist,  there  are  eight 
possible  combinations  of  those  variables.  A  general  logical  function  of 
three  variables  is  given  by: 


D  =  K  ABC  +  K  ABC  +  ABC  +  K  ABC  + 
1  2  o  *+ 


K  ABC  +  K  ABC  +  K_,  ABC  +  K0  ABC 
5  6  7  8 
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where  the  bar  over  a  letter  (e.g.,  C)  represents  "not  C," 
is  equal  to  1  or  0  depending  on  whether  the  corresponding 
combination  is  or  is  not  included  in  the  logical  function,  and 
"+"  represents  the  logical  "OR"  operaTon. 

For  e>  'mple,  if  is  eaual  to  l  and  all  remaining 
K's  are  set  equal  to  0,  then  the  function  specified  is: 

D  =  ABC 

In  another  example,  with  K  and  K0  set  equal  f..  1,  and  the  re¬ 

maining  K's  set  equal  to  0,  the  function  is: 

D  =  ABC  +  ABC 

which  is  read:  "D  equals  A  and  B  and  C  or  A  and  B 
and  not  C"  which  simplifies  to: 


D  =  AB 


A  logical  function  can  be  specified  in  two  ways. 
One  is  at  a  general  level  in  which  the  logical  function  is 

simply  written  out;  e.g., 


D  =  AB 


In  the  second,  the  function  is  specified  by  giving  example 
solutions,  such  as  shown  in  Table  1. 


Table  1 


Example  Logical  Specification 


I 


Here  we  have  two  example  solut.  -os  which  state  that  when  A  and 
B  and  C  are  true,  condition  D  is  true;  and  also,  when  A  and 
B  and  not  C  are  true  (i.e.,  C  is  false),  condition  D  is  also 
true.  A  complete  -  pec:f'ication  of  a  three-variable  equation 
requires  eight  independent  solutions  such  as  shown  in  Table  2.  If, 
however,  an  associated  computer  processor  wore  to  generalize 
by  the  rule  "The  output  is  always  lalse  unless  specir:ed 
as  being  true,"  the  twi  'xample  solutions  provided  in  Table  1 
would  be  sufficient  to  -  >mpletely  specify  the  necessary  logical 
function. 

With  the  first  method  discussed  above,  the  problem  solu¬ 
tion  is  complete  V  specified  by  the'  formula.  With  the  second 
method,  the  pro  i  solution  is  implicitly  specified  by  example 
solutions.  Z  tncr  method  can  be  used  to  communicate  the 
solution.  'he  j' ’ty  with  specifying  general  solutions  is 
that,  as  the  ft r . t ion  oecomes  increasingly  more  complicated,  it 
becomes  >  'ort  difficult  t.  i^erstand  and  rectify.  As 
a  result,  the  user  wil’  often  covertly  generate  for  his/her 
own  use,  a  number  of  example  sole'  ions  which  may 
then  be  employed  to  verity  the  general  solution.  This  process 
often  occurs  in  testing  a  computer  program. 


Table  2 

Specif icat ion  of  Logical  Function 


The  advantage  rovided  by  a  general  solution  is  that  it 
is  a  compact  way  of  representing  and  communicating  the  solution. 
The  advantage  of  specifying  solutions  by  means  of  examples  is 
that  they  are  easily  uncerstood  and  checked.  Their  disadvantage 
is  that,  as  the  number  of  variables  (N)  become  large,  the 
number  of  example  solutions  required  to  completely  specify  the 
logical  function  increases  by  2  . 

Dealing  with  logical  problems  of  this  type  is  believed  to 
be  a  fundamental  difficulty  not  only  in  specifying  problem  solu¬ 
tions  for  comp  ter  programs,  but  also  in  many  other  problem 
areas  where  there  are  a  large  number  of  variables  to  be 
considered  and  where  each  can  be  of  equal  importance.  These 
problem  areas  include  specification  of  instructions  for  virtually 
any  purpose  (e.g.,  instructing  individuals  in  military,  industrial, 
and  home  situations),  including  instructions  for  developing 
solutions  for  computer  implementation  as  well  as  for  using 
computer  systems . 


Work  Completed 


The  portion  of  the  ongoing  research  completed  and  reported 
here  consists  of  results  obtained  for  two  participant  categories,  each 
working  with  three  levels  of  problem  complexity,  and  three  levels 
of  processor  complexity. 

The  experimental  task  required  that  the  participant 
enter  example  solutions  into  a  computer.  The  computer  then 
examined  those  example  solutions,  and  formed  a  general  solution. 
This  general  solution  was  refined  by  the  computer  as  the  participant 
entered  each  new  examole  solution.  The  participant  was  able 
to  check  the  computer's  general  solution  by: 

1  .  requesting  and  examining  example  solutions 
generated  by  the  computer  in  accordance 
with  its  general  solution,  and 

2.  by  examining  the  computer's  general  solution 
function. 
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The  Task 


The  experimental  task  employed  in  this  study  re¬ 
quired  specification  of  a  logic  for  the  selection  of  a  hypothetical 
Navy  task  force.  The  task  involved  choosing  ships  from 
a  ship  list  which  identified  tne  ship  type,  the  transiting  time 
(.e.,  time  required  for  the  ship  to  get  from  its  present 
position  to  the  desired  site),  and  stationing  time  (i.e.,  the 
number  of  days  the  ship  could  remain  on  station  with 
available  provisions).  The  participant  was  required  to 
specify  example  ship  combinations  for  each  problem 
worked.  This  ranged  from  the  simple  task  of  soccify- 
ing  three  different  ship  mbinations  in  the  first  proolem, 
to  additional,  more  difficult  combination  s  in  later  problems. 

In  addition  m  this  specification  of  ship  combinations, 
the  participant  also  had  to  specify  by  example  solutions 
the  range  of  transiting  and  stationing  times  required.  For 
instance,  if  the  required  transiting  time  was  10  days  or 
less,  the  participant  had  to  search  the  ship  list  to  find 
one  example  ship  of  the  proper  type  that  would  establish  the 
desired  upper  limit  on  transiting  time.  Aim,  the  participant 
had  to  search  the  list  to  find  another  ship  of  the  proper 
type  which  had  a  transiting  time  that  established  the  desired 
lower  limit.  Thus,  the  participant's  task  was  not  only 
to  establish  the  proper  combinations  of  ship  types,  but 
also,  for  each  ship  and  combm  tion  of  ships,  to  establish 
by  example  solutions  the  desired  range  of  transiting  and 
stationing  times . 

Each  articipant  was  also  required  to  deal  with 
various  processor  complexity  levels.  He/she  did  not  have 
previous  knowledge  of  how  a  particular  processor  (computer) 
would  interpret  the  example  solutions  provided .  For  instance,  at 
one  complexity  level,  the  processor  would  generalize  transiting 
and  stationing  times  over  particular  ship  types  only.  At  another 
processor  complexity  '"vel,  the  processor  would  generalize 
transiting  and  stations  ■  times  over  all  ship  types  in  a  particular 
combination. 
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The  participant  evaluated  the  general  solution 
(processor  ship  selection  logic  [SSL])  produced  by  the 
processor  to  make  sure  it  was  a  correct  solution.  If  it  was 
incorrect,  the  participant  modified  it  by  providing  additional 
example  solutions.  The  participant  could  also  evaluate  the 
processor's  general  solution  by  asking  the  computer  to 
generate  its  own  example  solutions  and  then  by  evaluating  the 
newly  generated  solutions  for  accuracy  and  completeness. 

Design  of  the  Experiment 

The  experiment  usi  d  a  repeated  measures  Latin 
Square  design  (Plan  #9  cited  in  Winer,  1971,  pp.  727  -  736). 
The  design  is  shown  in  Table  3.  The  factors  investigated 

were: 

a.  3  levels  of  problem  complexity,*  as  measured 
by  Halstead's  L  Metric,  where  each  level  re¬ 
quired  a  ^.fferenl  amount  of  effort  to  correctly 
specify  the  problem  solution. 

b.  3  levels  of  processor  complexity*  where  each 
level  required  the*-  a  different  amount  of  inrerma- 
tion  be  provkf  by  the  user  to  correctly  specify 
the  solution. 

c.  2  participant  categories:  bookkeepers/accountants 
and  expert  comouter  programmers. 

Each  factor  was  fixed,  but  the  group  (G)  and  the  'articipants 
within  each  group  were  random  factors.  (An  additional  factor, 
various  levels  of  feedback  aids,  will  be  .  vialyzed  in  later  ex¬ 
periments.)  The  order  ot  presentation  lor  each  group  (G) 
was  randomized,  resulting  in  the  presentation  schedule  shown 
in  Table  4  for  programmers  and  in  Table  5  for  bookkeepers/ 
accountants . 


Measures  of  problem  complexity  and  processor  complexity 
are  discussed  in  subsequent  sections  of  this  report. 
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Table  3 

Schematic  for  Experiment  Plan 


Programmers 


G1 

A1B2 

A 

2  3 

A  B 

3  1 

Go 

A  B 

1  1 

A  B 

2  ° 

A  B 

3  3 

o 

A  B 

1  3 

j 

A  B 

3  2 

Bookkeepers/Accountants 

G4 

A  B 

1  2 

A  B 

2  3 

A3B1 

°5 

A  B 

1  1 

A  B 

2  2 

A  B 

3  3 

G6 

A  be 
i  1  3 

A2B1 

A  3^ 

3  2 

A.  is  a  level  of  problem  complexity. 

B  is  a  level  of  processor  complexity . 

j 

Each  combination  of  A.  and  B.  is  an 
experiment  cell .  1 

C  is  a  group  of  10  participants, 
k 


Tab’e  4 

Computer  Programmers 
Problem/Processor  Order 


Session  # 


Participant  # 

2 

3 

4 

Group 

Order 

Problem/ proces 

our 

14 

31/1 

15/2 

52/3 

1 

5 

15 

3i/3 

15/1 

52/2 

2 

5 

16 

31/2 

15/3 

52/1 

3 

5 

17 

52/y 

31/1 

1  5/2 

1 

4 

18 

52/2 

31/3 

15/1 

2 

4 

19 

52/1 

31/2 

1  5/3 

3 

4 

20 

1  73 

52/3 

31/1 

1 

-1 

21 

15/1 

52/2 

31/3 

2 

1 

22 

15/3 

52/1 

31/2 

3 

1 

23 

31/1 

52/3 

15/2 

1 

6 

24 

31/3 

52/2 

15/1 

2 

6 

25 

31/2 

52/1 

15/3 

3 

6 

26 

52/3 

15/2 

31/1 

1 

3 

27 

52/2 

1  5/1 

31/3 

2 

3 

28 

52/1 

16/3 

31/2 

3 

3 

29 

15/2 

31/1 

52/3 

1 

2 

30 

15/1 

31 A 

52/2 

2 

2 

31 

1  5/3 

31/2 

52/i 

3 

2 

32 

15/2 

52/3 

31/1 

1 

1 

33 

15/1 

33/3 

31/3 

2 

1 

34 

15/3 

52/1 

31/2 

3 

1 

35 

15/2 

31/i 

52/3 

1 

2 

36 

1  5/1 

31/3 

52/1 

2 

2 

37 

i  /3 

31/2 

52/1 

3 

53/3 

31/1 

’  5/2 

1 

4 

39 

52/2 

31/ 

15/1 

2 

4 

40 

52/1 

3V2 

15/3 

3 

4 

41 

31/1 

52/3 

15/2 

1 

6 

42 

31/3 

62/2 

1  5/1 

2 

6 

43 

31/2 

52/i 

15/3 

3 

G 

Notes:  Problem 

numbers 

were  ass 

iuned  randomly  to 

conceal  the 

relative  complexity  of  the  oroblems .  They  :  #93,  #15, 

#52,  and  #31  in  order  of  increasing  complexity. 

Session  1  -  93/3  <j  ’roblcm  #93,  Processor  #3).  This  was 
the  ore-test.  !  ’anticipants  #1  -  13  were  used  in  the  Pilot 
I  est,  not  in  the  experiment. 


T able  5 

Bookkeepers/ Accountants 
Problem/^rocessor  Order 


Session  # 


Participant  # 

'> 

3 

4 

Group 

Order 

44 

31/1 

15/2 

52/3 

4 

5 

45 

31/3 

15/1 

52/2 

5 

5 

46 

31/2 

15/3 

52/1 

6 

5 

47 

62/y 

31/1 

15/2 

4 

‘  4 

48 

52/2 

31/3 

15/1 

5 

4 

49 

52/1 

31/2 

15/3 

6 

4 

50 

1  5/2 

52/3 

3V/1 

4 

1 

51 

15/1 

52/2 

31/3 

5 

1 

52 

15/3 

52/1 

31/2 

6 

1 

53 

31/1 

52/3 

1  5/2 

4 

6 

54 

31/3 

52/2 

•  15/1 

5 

6 

55 

31/2 

52/1 

'  15/3 

6 

6 

56 

52/3 

15/2 

31/1 

4 

3 

4 

57 

52/2 

15/1 

31/3 

5 

3 

58 

52/1 

j/3 

31/2 

6 

3 

59 

15/2 

31/1 

52/3 

4 

> 

60 

15/1 

31/3 

52/2 

5 

2 

61 

15/3 

31/2 

52/1 

6 

2 

62 

15/2 

52/3 

31/1 

4 

1 

63 

15/1 

52/2 

31/3 

5 

1 

64 

15/3 

52/1 

31/2 

6 

1 

65 

15/2 

31/1 

52/3 

4 

2 

66 

15/1 

31/3 

52/1 

5 

2 

67 

15/3 

31/2 

52/1 

6 

2 

68 

52/3 

31/1 

15/2 

4 

4 

69 

52/2 

31/3 

15/1 

5 

4 

70 

52/1 

31/2 

15/3 

6 

4 

71 

31/1 

52/3 

1  5/2 

4 

6 

72 

31/3 

52/2 

15/1 

5 

6 

73 

31/2 

52/1 

15/3 

6 

6 

Notes:  See  Table  4. 
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(D  CO  CO  CO  '  01  OJ 


Referring  to  Tables  4  and  5,  the  numbers  in  column  one 
are  the  numbers  assigned  to  the  participants.  The  numbers  in 
columns  two,  three,  and  four  refer  to  the  problem  number  and 
processor  level  number,  -  espectively;  e.g.,  31/1  =  Problem 
Number  31 ,  Processor  Number  1  .  The  numbers  in  column  five 
refer  to  the  groups  to  which  the  participants  were  assigned  and, 
in  column  six,  to  the  order  of  problem  presentation. 

Each  participant  was  given: 

1  .  a  description,  via  video  tape,  of  the  ex¬ 
periment  task  along  with  an  illustrative 
solution  to  the  task. 

2.  a  description,  via  video  tape,  of  the 
procedure  for  entering  data  into  the 
computer. 

3.  a  pre-test  problem  -  a  low-level  complexity 
problem  with  Processor  Level  Br  (least 
general  ization) . 

4.  Test  Problem  #X,  Processor  level  l. 

5.  Test  Problem  #Y ,  Processor  Level  m. 

6.  Test  Problem  #2,  Processor  Level  n. 

Note  that  values  for  X,  Y,  2,  l,  m,  n  were  taken  from  Tables 
4  or  5  in  accordance  with  the  participant's  group  number 

When  Step  3  above  was  completed,  the  performance 
score  on  the  pre-test  was  determined.  If  the  participant's 
input  example  solution  yielded  more  than  one  correct  ship 
combination  and  the  minimum/maximum  times  for  transiting 
and  stationing  respectively  were  not  equal  (these  two  conditions 
were  taken  as  evidence  that  the  participant  understood  the 
instructions),  the  participant  was  permitted  to  enter  the  experi¬ 
ment.  If  the  participant  did  not  input  example  solutions 
yielding  the  result  cited  above,  he/she  was  not  permitted  to 
enter  the  experiment. 
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Pilot  Test 


Eight  programmers  and  five  bookkeepers/accountants 
were  used  in  the  pilot  test  to  determine  suitable  levels  of 
problem  complexity.  The  time  required  for  presentation  of 
instructions  and  for  completion  of  each  problem  was  also 
determined.  Each  pilot  participant  received  the  test  instruc¬ 
tions  and  was  asked  to  solve  four  problems  of  different  levels 
of  complexity.  All  test  problems  in  the  pilot  test  used 
Processor  Level  B  . 

O 

Resul's  of  the  pilot  test  indicated  that: 


1  .  The  ship  'ist  should  be  removed  from  the 

computer.  Initially,  the  ship  list  was  presented 
on  the  computer  terminal,  but  it  was  found 
that  participants  continually  referred  back  to  it 
and  took  notes  about  suitable  ships.  This 
nroved  to  be  too  time  consuming,  so,  for  ref¬ 
erence,  a  typewritten  ropy  of  the  ship  list  was 
given  to  each  participant. 

2.  The  initial  problem  design  required  excessive 
time  and  had  to  be  revised  so  that  a  problem 
could  be  completed  in  one  hour. 

3.  Time  required  to  read  the  instructions  was 
excessive  in  som  cases.  Therefore,  it  was 
decided  to  pace  the  instructions  by  presenting 
them  simultaneously  via  video  tape  and  type¬ 
written  copy  -  ttie  participants  could  then 
follow  along  with  their  typewritten  copy. 


The  Experiment  Task 


The  experiment  task  was  described  to  the  participants 
as  follows: 

"In  order  to  investigate  the  ability  of  individuals  to 
develop  accurate  instruction. .  and  to  test  interpretations  of 
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their  instructions,  we  are  conducting  this  experiment  in  which 
you  will  be  presented  with  several  problems  of  various  levels 
of  difficulty  and  asked  to  develop  Instructions ,  in  the  form  of 
Example  Solutions  to  problems  that  we  specify.  You  will  also 
be  asked  to  enter  your  example  sc  lutions  into  a  computer 
terminal.  The  example  solutions  mat  you  enter  will  be 
interpreted  by  the  computer  as  Instructions  to  be  used  in 
solving  the  general  prob1'  for  which  your  inputs  are  example 
solutions . 

"Example  solutions  to  problems  are  like  specific 
instructions  which  can  be  interpreted  (or  generalized)  for  broad 
application.  Thus,  each  example  solution  is  part  of  a  specifica¬ 
tion  of  a  general  solution.  As  several  example  solutions  are 
developed  and  input  to  th«_  computer.  The  computer  will 
begin  to  form  a  general  solution.  In  order  to  evaluate  the 
computer's  general  solution  for  accuracy  and  completeness,  you 
must  test  it  by  commanding  that  the  computer  develop  its  own 
solutions  which  you  can  then  evaluate.  If  the  computer's 
general  solution  is  wrong  or  incomplete,  you  will  have  to 
develop  additional  example  solutions  to  show  the  computer 
where  it  was  wrong  in  its  general  solution.  This  iterative, 
interactive  process  will  continue  until  you  have  provided 
enough  example  solutions,  tested  the  computer  output,  and  are 
satisfied  that  the  general  solution  selected  by  the  computer  is  the 
proper  so'uiion.  During  this  process  you  will  be  provided 
with  feedback  aids  from  the  computer  to  assist  you  in  your 
task. 


''Illustrative;  Prob'nm  (A  pro .  .cm  and  solution  to  show 

you  b  w  it  is  done) 


Tht  Navy  has  -  n  ordered  to  be  ready  to  assemble  a 
task  force  for  an  extended  rri.ssion  in  fbe  Indian  Ocean.  In 
order  to  identify  possible  ships  for  this  task  force  wuen  the 
assembly  order  is  given,  Harry  Smith,  an  analyst  for  the  Navy, 
has  been  asked  to  spc.  ify  t  ic  instructions  necessary  to  scan  a 
file  of  available  ships,  to  apply  the  ship  requirements  necessary 
for  the  composition  of  the  task  force,  and  to  select  one  or  more 
"solutions"  (i.e.,  suitable  ets  of  ships)  for  the  missionv 

"Since  it  is  not  known  when  the  assembly  order  will 
be  given  and  the  list  of  available  shins  changes  from  time  to 
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time,  Harry's  job  is  not  merely  to  select  a  suitable  set  of 
ships  for  the  mission.  Instead,  Harry's  job  is  to  develop 
multiple  example  solutions  that  together  define  the  logic  (i.e., 
specify  all  combinations)  of  the  ship  selection  process.  The 
objective  is  to  speedy  the  logical  rules  for  selecting  ships;  then 
later,  when  the  ships  are  actually  to  be  identified  for  the 
mission,  the  logic  can  be  applied  to  the  list  of  available  ships 
and  the  appropriate  ships  selected  automatically  by  the  computer. 

"Harry's  fir  L  step  is  to  obtain  the  criteria  for  this 
task  force . 

1.  The  ships  needed  for  the  task  force  are: 

•  One  (1)  nuclear  attack  aircraft  carrier  (CVAN) 

•  Two  (2)  submarines  (SS) 

•  Two  (2)  destroyers  (DD) 

•  One  (1)  oiter  (AO) 

2.  The  present  position  of  the  ships  and  their  speed 
must  permit  arrival  at  the  assembly  point  within 
5  days . 

3.  Fuel  on  board  must  permit  stationing  without 
resupply  for  a  10  day  oeriod. 

"Harry's  next  step  is  to  construct  a  list  of  ships  which 
includes  the  necessary  data  on  available  ships.  He  constructs 

a  ship  list  instead  of  using  the  actual  ship  file  because  the  a< lal 

file  of  available  ships  may  not  con',  .an  all  combinations  of  ship 
types,  arrival  times,  an'4  fuel  loads  necessary  to  permit  seV 'Ction 
of  example  so'  'ons  specifying  the  necessary  combinations  v f 
ships  (i.e.,  necessary  to  specify  the  ship  selection  logic).  He 
developed  the  ship  list  shown  in  Table  7,  and  entered  it  into  the 
computer . " 

(land  ol  Dxperiment  1  ask  Description) 

*Note:  When  a  nuclear  ship  is  specified,  a  non-nuclear  ship 

is  not  accepta  likewise,  when  a  non-nuclear  ship 

is  specified,  a  nuclear  ship  is  not  acceptable. 


Problem  Variations 


Four  problems  of  different  complexifies  were  used  in 
the  experiment.  Complexity  is  a  function  of  the  number  of 
correct  solution  combuwe  uns  and  is  defined  in  the  next  section. 
The  problems  were  assigned  random  identification  numbers  so  as 
not  to  reveal  their  relative  complexity  to  the  participant.  One 
problem  (#93)  was  used  in  the  pre-test  an  1  the  other  three 
problems  were  used  in  the  experiment.  The  least  complex 
problem  (Problem  #93)  required  spe.  -fication  of  three  different 
ship  combinations  each  involving  tour  types  of  ships.  The  next 
level  of  problem  complexity  was  Problem  #15,  which  required 
six  ship  combinations  a<  tin  using  four  different  ship  types.  In 
order  of  increasing  c<"  '"xuy  were  Problem  #52,  wh'ch  required 
9  ship  combinations  each  u  .olviny  either  4  or  5  ship  types, 
and,  finally.  Problem  #31,  which  required  14  combinations 
each  using  5  ship  ty;  ..  The  four  ■  -'oblem  statements  are 
provided  in  Appendix  A . 

Measurement  of  Problem  Complexity 

Halstead's  E  M-lric  (Halstead,  1977)  is  a  measure  of 
the  mental  effort  (E)  (i.e.,  the  numbei  of  discriminations) 
required  to  accompliun  n  task.  The  m  •trio  perm;ts  decomposition 
of  a  task  into  a  number  if  elementary  discriminations  required 
by  the  task.  ’alstead's  L  tietric  is  frequently  used  in  software 
research.  It  permits  preparing  (and  in  some  cases  combining) 
results  obtained  by  different  research  programs .  Halstead's 


E  Metric  is  given  by: 

E  =  V2/V*  (1) 

where:  V  is  th*>  "program  volume"  and  V*  is  the  'fcotenha! 

program  volume."  V  is  -;ven  by: 

V  =  N  logQ  n  (2) 

where:  N  =  i  (3) 

n  -  n  +  n  (4) 


N  —  The  total  number  of  operators 

N  -  The  total  number  of  operands 

n^  -  The  un  rue  operator  count 

n 

2 


=  The  unique  operand  count 


V*  is  given  by: 


V*  = 

n*  log  n* 

2 

(5) 

where:  n*  = 

n  *  +  n  *  =  2  +  n  * 

12  2 

(0) 

n1*  = 

The  potential  operator  count 

n2*  = 

The  potential  operand  count. 

According  to  Halstead,  the  minimum  possible  number  of  operv'.ors 
n^*  for  any  algorithm  is  2,  one  operator  for  the  function  name  and 
another  for  its  assignment.  Thus  n  *  -  2. 

Furthermor«  ,  n  *  is  evalum  1  ‘he  minimum  number  of 
different  input/output  parameters.  The  quantity  N  in  equation  (3)  is 
called  the  "program  let  th;"  the  quantity  n  in  equation  (4)  is  the 
"vocabulary  size;"  and  the  quantity  n*  in  equation  (6)  is  the  "potential 
vocabulary  size."  All  of  these  quantifies  are  abstract  and  may  be 
difficult  to  visualize . 

In  order  to  compute  the  E  me.  ic  for  this  experiment,  the 
problem  solutions  were  written  in  explicit  logical  form  and  the 
operators  and  operands  were  counted.  Thus,  for  instance,  the  solution 
to  Problem  #93  was  written: 

(CVA  and  CVA  and  SS  and  SS) 
or 

(CVAN  and  CVAN  and  SS  and  SS) 
or 

(CVA  and  CVAN  and  SS  and  SS) 

wher., ;  CVA  is  an  aircraft  carrier  non  nuclear 

CVAN  is  an  aircraft  carrier  nuclear 

SS  is  a  submarine. 


r 


The  total  number  of  operands  (N  )  =  12 

i.e.,  there  are  3  CVA's,  3  CVAN's  and  6  SS's  in  the 
solution . 

n^*  =  2  as  indicated  previously, 

n^*  =  3,  i.e.,  CV A,  CVAN,  SS  are  the  three 

potential  input/ output  parameters 

The  calculation  of  Halstead's  E  metric  for  each  problem  is  given  in 
Table  6 . 

The  Experiment  u,-~ocessors 

Eased  on  the  examp, e  solutions  input  into  the  computer  by 
the  participants,  the  procc  sor  (computer)  is  able  to  form  a  ship 
selection  logic  (SSL)  by  which  the  comuuter  can  select  ships  in 
a  potential  task  force.  The  SSL  consists  of: 

a.  set  of  ship  combinations, 

b.  minimum/ maximum  (MIN/MAX)  values  for  transit 
times  for  each  ship  type  in  each  set  of  ship  combinations, 

c.  M1N/MAX  values  !ur  stationing  times  for  each  ship 
type  in  each  sot  of  ship  combinations. 

MIN/MAX  values  for  transit  ...  toning  times  were  a  function  of 

the  processor  generalisation  capability  (i.e.  ,  processor  complexity). 
Tfie  three  complexity  lev-  •'a;  employed  in  this  study  were: 

Level  B  ■  Thi>  M"\'  and  MAX  transit  and  stationing  times 

v.  ./  to  ill  ships  r  rdless  of  type  or  combina¬ 
tion  . 

Level  To  MIX  and  MAX  mansit  and  stationing  times 

apply  to  all  ships  u <  -ach  particular  combination. 

Level  H  :  The  M.IN  and  MAX  transit  and  stationing  times 
apply  to  each  ship  type  in  each  combination. 

A  solution  to  an  ’  lustrative  problem  showing  the  effect  on  the  SSL  of 
each  level  of  processor  complexity  is  presented  in  Tables  8A  to  8D. 
i  sole  7  is  the  list  of  ships  used  in  the  illustrative  problem. 

Table  bA  provides  the  problem  statement  and  four  ex¬ 
ample  .a  •'  .tions  that  are  assumed  to  have  been  entered  into 
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Table  7 


List  of  Shins 


Fuel  Supply 


Shir-  it 

Ship  1! 

1  rans i t 

required 

l  i me  on  Station 

(Days)  C-  ays) 

1 

CVA 

5 

10 

i 

CVA 

4 

JO 

3 

CVAN 

b 

2C  J 

4 

CVAN 

b 

100 

5 

CA 

4 

3 

t> 

CA 

7 

1  r\ 

7 

CC 

4 

1  0 

B 

cc 

B 

20 

O 

CON 

b 

100 

10 

DO 

b 

10 

1  1 

DD 

b 

5 

14 

DO 

l ; 

10 

13 

i  -  ) 

4 

12 

14 

DD 

7 

10 

1  5 

DO 

y 

1  o 

1  0 

SS 

5 

7 

1  , 

ss 

b 

J0 

IB 

SS 

4 

10 

'  0 

SS 

B 

12 

JO 

ss 

12 

J1 

'  SN 

b 

1  B0 

JJ 

SSN 

6 

200 

23 

SSN 

4 

250 

34 

AO 

20 

Jb 

AO 

b 

30 

26 

AO 

7 

Legend: 

20 

CVA 

Attack  Aircraft 

Carrier  CON 

Guided  Missile  Cruiser 

CVAN 

Attack  Aircraft  Carrier 
(Nuclear  Propuls  ion)  DD 

(Nuclear’  Propulsion) 
Destroyer 

CA 

1  leavy  Cruiser 

SS 

Submarine 

CO 

AO 

Guided  Missile 

Oiler 

Cruiser  SSN 

Submarine  (Nuclear  Propul 
sion) 
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Table  BA 


An  Illustration  of  the  • !  ect  of 
The  Three  1  >'Qce»aor  Levels  of  Cm  -  'plex  a. 


The  criteria  >r  the  task  !  >rce  are: 

1.  r  e  ships  needed  for  the  task  force  are: 

•  One  (1)  nuclear  attack  a  •.••craft  carrier 
(CVANi  or  attack  aircrati  carrier  (CVA) 

•  Two  (2)  submarines  (sS) 

•  Two  (2)  destroyer  (UU) 

•  One  (0  oiler  (AO) 

The  present  po  men  of  1  ie  ships  and  their 

speed  must  permit  arrival  at  the  assembly 
unt  witlun  5  days. 

3.  Fuel  on  board  mist  permit  stationing  w.'.hout 
resupply  for  a  fO  day  period. 


These  problem  criteria  specify  two  distinct  combinations 
of  siiip-j,  as  follows: 

Combination  1:  CVAN,  SS,  SS,  DD,  OD,  AO 

Combination  2:  CVA,  SS,  SS ,  DD,  DD,  AO 


i  aule  BA  (l  oncluded) 


An  Illustration  of  the  Effect  of 
The  Three  Processor  Levels  of  Complexity 


Suppose 

four  example 

solutions 

are  entered  as 

follows: 

Example 

T rai  i-D  T ime 

Fue.  Supply  on 

Solution 

Ship  # 

Ship  ype 

Requiredfdays 

.)  Station  (days) 

1 : 

1 

OVA 

5 

!  0 

1  7 

SS 

o 

1  0 

20 

oS 

o 

12 

10 

DD 

5 

10 

13 

DO 

4 

1  2 

24 

AO 

4 

j 

o . 

2 

CVA 

a 

_0 

17 

‘ 1 S 

5 

1  0 

20 

oS 

p 

12 

10 

fv 

• ; 

i  0 

13 

Do 

4 

12 

25 

AO 

30 

3: 

3 

CVAN 

5 

200 

17 

-S 

5 

1  0 

20 

S_.. 

3 

12 

10 

DD 

5 

10 

13 

DO 

4 

12 

24 

AO 

4 

20 

. 

CVAN 

5 

100 

17 

SS 

o 

10 

0 

SS 

1  2 

10 

DD 

5 

13 

DD 

4 

!  2 

25 

AO 

■■ 

30 

r 

> 


;L  Ft '"Tied  by  Proce-  o"  -eve  A 


rab'.  •  01  ’ 

i  ovc-  it  for  '-be  Illustrative  Problem 


~  o’  S'""' 

! "e. 

>1!'*  V  *  >0 

’  ransit 
\ N  VAN 

tat 

V.N 

ioninc 

MAN 

Combi"  ‘.son  ’ 

1 

OVA 

3  b 

10 

200 

'  -  1 

10 

200 

1'  '» 

3  ;.> 

10 

200 

1 

At' 

10 

200 

Combination  .1 

1 

CVAS 

3 

10 

20C 

Table  SC 


SSL  Formed  by  Proce  ■  : 


.or  Level  Bo  for  the  Illustrative  Problem 


#  of  Shies 
Required 

Ship  T yoe 

T  ransit 

MIN  MAX 

Stationing 
MIN  Mi  AX 

Combination  1 

1 

CVA 

3 

5 

10 

30 

2 

SS 

3 

5 

10 

30 

2 

CD 

3 

5 

<0 

30 

1 

AO 

3 

5 

10 

30 

Combination  2 

1 

CVAN 

3 

5 

10 

200 

o 

SS 

3 

5 

10 

200 

2 

CL) 

3 

5 

10 

200 

1 

AO 

3 

5 

10 

200 

i 


30 


Table  8D 


SSL  Formed  by  Processor  Level  3^  for  the  Illustrative  Problem 


#  of  Ships  T ransit  Stationing 


Required 

Ship  Type 

MIN 

MAX 

MIN 

MAX 

Combination  1 

1 

CVA 

4 

5 

10 

20 

2 

SS 

3 

5 

10 

12 

2 

DD 

4 

5 

10 

12 

1 

AO 

5 

20 

30 

Combination  2 

1 

CVAN 

5 

5 

100 

200 

2 

SS 

3 

5 

10 

12 

2 

DD 

4 

5 

10 

12 

1 

AO 

5 

20 

30 

Note:  The  range  of  M1N/MA X  v.’ues  are  limited  by  the  available 

ships  on  the  shiD  list. 

*  Numbers  correspond  to  ships  <_n  list  given  in  Table  7. 


the  computer.  Table  8B  is  the  SSL  formed  by  Processor  Level 
B1  from  the  four  example  solution  .  At  Processor  Level  the 
processor  scans  the  transiting  and  stationing  times  of  all  ships 
in  Tl_  example  solution.,  and  determines  the  MIN  and  MAX  values 
of  the  transiting  time  and  of  the  stationina  time.  These  MIN/ 
MAX  values  are  then  assigned  to  each  tmp  type  in  the  SSL  as 
shown  in  Table  8B.  For  instance,  the  MAX  station. ng  time  is 
the  same  (200  days)  for  all  ship  tyoes  in  both  ship  combinations . 

Processor  Level  3  generalizes  over  each  ship  combina¬ 
tion  (Table  8A)  instead  of  oVer  all  ship  combinations  as  does 

Processor  Level  B  .  This  restricted  generalization  is  illustrated 

1 

in  Table  8C  where  the  MAX  stationina  time  is  different  for  the 
two  ship  combinations .  The  differc-  ,e  occurs  because  the 
ship  with  the  greatest  stat.omng  time  (AO)  in  Example  Solutions 
1  and  2,  which  define  ship  combination  1,  has  a  stationing  time 
of  30  days.  In  contrast,  the  corresponding  MAX  stationing  time 
in  Example  Solutions  3  and  4  is  200  days  for  a  CVAN . 

Finally,  Processor  Level  B  (Table  8D)  generalizes  only 
over  each  ship  type,  and,  as  a  result,  the  MIN  on  MAX  tran¬ 
siting  and  stationing  times  can  easily  be  different  Fr  each  ship 
type . 


The  impact  of  the  various  levels  of  Processor  generalization 
(complexity)  on  the  user  is  that  fewer  example  solutions  may  be  re¬ 
quired  to  establish  a  broad  range  of  MIN  and  MAX  transiting  and 
stationing  times  in  the  SSL  with  Level  !3  than  with  Level  B  .  In 
turn,  Levels  B^  and  B^  each  require  less  information  from  the  user 

than  does  Processor  Level  B  . 

3 

In  contrast,  while  the  amount  of  information  required  from 
the  user  increases  with  Processor  Level  ,  SQ,  and  B^,  in  that  order, 
the  amount  of  processing  required  decreases  with  that  order.  This  is 
because  the  processor  must  make  many  comparisons  of  MIN  and 
MAX  tra'  siting  and  stationing  time  values  for  Level  B  ,  fewer 
comparisons  for  Level  B2»  and  the  least  number  of  cemparisons  for 
Level  Bg.  Increasing  the  computer  calculation  load  decreases  the 
amount  of  information  required  from  the  user.  As  a  result, 
processor  'omplexity  can  be  calculated  from  two  viewpoints:  the 
user'  ’  and  the  processor's.  These  processor  complexity  measures 
are  discussed  in  the  next  two  sections . 
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Measurement  of  Processor  Complexity  (User's  Viewpoint) 

Proc  'ssor  complexity,  from  the  user's  viewpoint,  is  a 
function  of  the  amount  of  information  required  to  specify  the  SSL 
for  the  comouter.  “"his  amount  is  a  direct  function  of  the  level 
of  processor  complexity  and  is  computed  in  the  following  paragraphs 
for  the  oroblem  requiring  the  greatest  amount  of  information. 

Variables  to  b  specified  by  the  user  are: 

a.  The  number  and  type  of  ships  in  each  combir  ition, 
i.e.,  each  example  solution. 

b.  The  M’ M/MAX  stationing  and  transiting  times  re¬ 
quired  by  the  processor  level. 

Since  the  processors  at  each  specified  level  must  have  sufficient 
storage  to  permit  processing  of  all  problems,  the  problems  were 
examined  to  determine  the  maximum  storage  requ'^ed  for  each 
of  the  two  items  listed  above. 

Consider  first  the  amoi  nt  o  '  information  required  to 
specify  the  number  and  type  of  ships.  M.ne  types  of  ships  (see 
legend  to  Table  7)  are  available  for  selection.  Adding  "1"  to 
account  for  a  specification  of  "MO"  shin  -'esults  in  a  total  of 
10  required  items,  from  which  1  is  selected  by  the  user.  Ten 
items  require  3.322  bits;  however,  since  the  computer  must 
deal  with  distinct  levels  of  storage,  this  value  was  rounded  up 
to  4  bits. 

The  maximum  number  of  ships  per  combine" ;on  (whi-'h 
occurs  in  Problem  #  132)  is  10.  T’nus,  4  X  io  =  ao  bits  are  re¬ 
quired  to  store  data  foe  each  combination.  (Mote  that  some 
efficiency  could  be  obtained  by  racking  the  data,  since  4  bits  have 
been  allowed,  but  that  because  this  number  is  constant  for  all  processor 
levels,  this  efficiency  was  not  used.) 

The  maximum  number  of  ship  combinations  (from  Prob¬ 
lem  #31)  is  14.  T^us  the  storage  required  for  all  combinations 
of  ships  was  4  X  10  X  14  =  560  bits. 

Consider  next  the  amount  of  infi  mation  required  to 
specify  the  MIM/MAX  stationing  and  transiting  times.  This 
amount  is  a  runctlon  or  the  processor  level  and  is  computed  as 
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follows:  since  the  allowable  transiting  times  are  integers 

wunin  the  limits  0  -  1  5  days,  4  bits  each  (4  bits  permit 
discrimination  •  c  1 G  it<  ms)  cme  issianed  tc  specification  of 
M'd  and  MM  stationin'-’  tim-  s.  Thus,  specification  of  MIX/ 

MAX  “•"ansi ting  values  -equmes  3  bits.  Another  16  bits  are 
reauired  to  snec'cy  the  M’X/M.AX  values  for  stationing  times, 
yielding  a  grand  tal  -4  bits  oer  specification  for  a  set  or 
M IN/M. AX  values.  Calculations  for  each  processor  level  are 
as  follows: 

Level  Li  requires  one  specification  of  M1M/MAX 
values  oer  problem;  thus,  toe  number  of  bits 
requmes  -  64. 

Level  Li  roou-mes  one  specification  of  M IN/MAX 
values  per  .  -nbmai .on,  times  the  maximum 
number  -  'mblna*.  -n-v.;  t  -us  the  number  of 
bits  requmr.'  -  .’4  n.  63'.' . 

Level  Li,  require:.  one  specif ication  of  M IN/MAX 
values  per  *  imos  •>  maximum 

number  o<  ocvb-u'.anv.,  ‘  :m._.y  the  maximum; 
number  or  .  -vr  combination;  thus,  the 

number  <V  b  ts  rey.imoi!  =  34  X  14  X  10  =  3360. 

Thus,  the  amour'  of  n'or-n  at  ton  required  to  form  the  SSL  is  560 
bits  (which  is  the  max-mam  -itoracv  required  for  all  the  ship 
combinations  and  is  toe  same  for  .  ’!  nroces  .or  levels)  plus 
the  varying  amounts  or  i- >!  ormet  ion  v given  above)  required  to 
specify  the  limits  on  transitin’;  and  stationing  times. 

Processor  Complexity  T  ’»  'Ci.-i-.sor  >,  ewnoto*T 

The  processor  m  -  h,.ve  ‘me  'i\-mory  capacity  to 
store  all  information  required  to  .needy  -me  maximum  amount 
of  ship  data  as  well  as  t -e  qrea lest  variety  or  MIX/ M..AX  transiting 
and  stat. union  times.  'T;’us1  toe  processor  ror  each  level  of 
complev,ty  uses  563  b - 1- .  to  store  ‘me  •'  •*  it  a  and  3360  bits  to 
store  the  MIN/MAX  transiting  and  s’.mon;:  _  time  data  (based 
on  ‘■oe  requirements  for  the  or  ub’.e-m  -uirina  the  largest  amount 
of  storage). 
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In  addition  -e  r*->i---ory  storage  requirements, 

computer  processing  tan..  is  reouireu  to  perform  the  necessary 
comparison  or  ship  tr.  .’ting  and  station. ng  '-ones.  These 
comparisons  are  -vce  •  ary  .o  ■*> -term me  iV\ I N!/ VAX  transiting 
and  state  "ng  tim.  >s  and,  .  i-  .ir  '"d  previously,  the  number  of 
comparisons  is  a  *'  'c'.i'.n  of  processor  complexity  level. 

1  ne  number  of  repaired  eompar’sons  may  be  determined 
by  assuming  that  the  us.  •"  enters  minimum  number  of  example 

solut  ons  recumed  to  establish  tot*  aw  selection  logic  (SSL). 

'f  the  user  enters  ice  Mona’,  example  lotions,  additional 
comparisons  ame  re  -pence ,  :  sr  1  n(  •  ..  ,  m*  not  coniiidc^cd  m  the 

following  ..ii  .  i .  ,  . 1  . 

Levs'  1  '•  i  .'">•••,  m' •  .nn  ni  •  r\i*  »•  at  mo  M'nes  of  all 

.  pt;>: ,  s-  .  *.<  v  •  >..•  VAX  '  m  ;0:  ■ 1  fit"e.  X  there  are 

X  s.v.n  ct  '•  •  r.s  *  i..i.  and  'in  average  of  K  ,  i'nips 

per  cn|" i ■  M'.un  rf  -uined  ‘  or  .plot  on  to  o.  problem, 
•J'fr'  t  ie  •  e.imb..-'  n<  ■  iiP'  X  i  X  , .  T'-'js  K  K,-1 

on'  pare  ■  ■  ■  ■  inn  .  ■  .,j  si  f < .-.j  fie  largest 

•np.n  ;1.  A  .  p.r  •  .  or  comparisons 

pi>  n*.  p 1  '"'i  ■ .  si  a  ■  ■  on*  *  *  ’.  no  \  n  ,  p.  s  d  VAX  and 

V.l\'  sta.'..'ii . i’"c  .  ’’  a  ‘ota.  •.<*'  --(K,  K  -  1) 

coiii'M*'  ■  '■"*  *■  'i '  i^'‘p.  >•■><.  ■  e  ,i'vi.''.  .  processor • 

Level  id.  ■  recratno  ■  comparison  of  •  e  t nans i tiny  times  of  all 
id  up--.  >n  s  c.w"b  >o.  'has  (X  -  1  i  comparisons 

a, re  npc. . .  ''pn  ,  •  ,ph  combination.  '  •"*  X  combine- 

1  i *  . ,  ;s  s  -  4 )  c  wv»*'an>son.  an*,  nocuircd.  A 

a.'Tv'ar  rv.  s-hnn  o<~  cin'iw'wuris  a*  e  reauired  to 
de'.ennv'e  \"\  tei'v,".,  a*vd  VAX  and  V!\  stationing. 

'  has.,  •  v  •  ■  ■  i 1  of  con  ou*".sons  necumed  is 

.1 X  f-\  ~  '  X 


Level  n.  ....a  n.  • 

-  - k— 


ift  V  x 


Partin’  :  n  t  ■ . 


The  two  par*. ic 
programmers,  and  ex 
expert  programmers . 


>”enced 
elect'1  *n 


or;  as  consisted  of  i  •  ■  computer 
.K*oXXot *: >er s/account, v  woo  were  not 

criteria  for  establishing  whether 
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a  programmer  was  considered  an  expert  included:  work  with 
multi  o’  •  languages,  production  of  at  ’.east  10,000  lines  of  code, 
and  experience  with  multiple  computer  systems.  Selection 
criteria  for  the  non-programming  category  (bookkeepers/ 
accountants)  included  4  years  of  schooling  or  experience  in  the 
bookkeeping  field  ..md  no  extensive  exDerience  as  a  com’  ..'ter 
programme-'-'  Complete  selection  criteria  for  both  participant 
categories  are  given  a.  Appendix  B. 

Participants  were  obtained  by  commercial  personnel 
organizations  which  specialize  placing  programmers  and 

bookkeepers/accounf nnts  in  temporary  and  permanent  positions. 
Initial  screening  of  participants  was  d  .■  by  the  personnel 
agencies  in  accordance  with  the  selection  criteria  provid  'd  them 
(see  Appendix  B).  In  addition,  due  to  a  low  flow  rate  of  partici¬ 
pants,  particularly  in  the  bookkeeping/accounting  area,  an 
advertisement  was  placed  in  a  local  newsoaoer  stating  that  an 
evaluation  of  computer  equipment  was  brim-  .'-inducted. 

Twenty-six  people  answere  this  ad,  of  woich  22  wer  fou  -d 
to  be  acceptable.  A  copy  of  the  ad  is  given  in  Appendix  C. 

A  total  of  7  5  people  participated  in  the  experiment. 

Of  the  75,  1 5  were  not  used  because  'ey  did  not  achieve  an 
entry  level  criterion  score  on  the  pr<— ‘est  problem  (#93). 

Also,  13  additional  people  participate  >n’y  in  the  phot  test  onior  to 
the  experiment.  All  participants  were  over  2  ye<.,-  s  of  aye. 

Participants  who  were  obtained  through  agencies  (all 
computer  orogrammers  and  eight  of  the  bookkeeoers/accoumants) 
were  paid  in  accordance  with  the  polic  of  too  ngermos  woo 
obtained  them.  The  bookkeepers/accountants  w  were  obtained 
via  the  newspaper  adver*- isomer*,  w >ro  paid  $7.50  per  uour. 


Procedure 


Participa*  s  were  ...  eduled  ‘  ~  other  a  morninc  session 
beginning  at  8:00  or  an  afternoon  session  be  pi  on  mo  at  1 : 00 . 

When  a  parties  ot  arrived,  he/she  was  greeted  by  r'"nr,e,  offered 
a  refreshment  and  asked  to  fill  >  t  a  olograph  ical  cuestlon- 
naire  (the  questionnaire  is  given  in  Appendix  O)  to  verify 
that  the  participant’s  experience  satisfied  the  experiment 
entrance  criteria  and  to  outain  additional  information  regarding 
the  level  of  experience  in  his/her  particular  field.  !f  the 
participant's  experience  did  not  satisfy  the  criteria,  hc/she 
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was  not  usee’  in  the  exner.ment.  !r  "he  oarticipant  was  accept¬ 
able,  the  experiment  was  briefly  explainec,  and  the  participant 
was  provided  with  a  consent  form  (Aocendix  E)  having  been 
assured  that  no  personal  risk  was  involved.  ~rhe  participant 
then  signed  this  form  to  indicate  that  he/s  he  nderstood  these 
arrangements . 

The  participant  was  next  seated  in  the  experiment 
room.  The  room  was  approximately  12  X  IS  feet  in  size, 
with  a  video  tape  recorder  and  video  monitor  located  on  one 
table,  and  the  computer  and  terminal  on  a  separate  table. 
Participants  were  asked  to  make  themseleves  comfortable  and 
to  adjust  the  light  and  ventilation  to  them  ...israction . 

Instructions  for  the  experiment  were  presented  two 
oarts,  both  of  which  were  on  video  tape.  The  first  part 
described  the  experiment  problem  and  gave  a  method  for  solving 
the  problem  including  an  example  solution.  (T  -ese  Instructions 
have  been  discussed  earlier  in  me  sn~tion  <:•  titled  ''Experi¬ 
mental  Problem.")  The  second  part  gave  .nstructions  on  how 
to  enter  data  into  the  compute^  -  and  also  included  an  illustrative 
problem  solution.  Since  this  pomion  of  the  instruction  employed 
a  dynamic  display  of  the  operation  of  the  computer,  it  cannot 
be  presented  here. 

After  f  instructions  were  presented,  the  participant 
was  seated  in  front  of  the  computer  terminal.  He/she  was  asked 
to  use  the  manual  key  pad  (consisting  of  keys  tabled  0-9  and 
ENTER).  The  participant  was  asked  to  enter  example  solutions 
using  numbers  that  corresponded  to  a  ship  list  (An  examp’e  is 
shown  in  Appendix  F)  containing  various  types  of  ships  am  their 
transiting  and  stationing  times.  The  participants  could  refer  to 
this  list  at  any  time  during  the  experir  «_ot.  In  addition,  a  pad 
of  paper  and  pencils  were  provided  for  notes,  calculations,  etc. 
These  sheets  were  kept  in  each  participant's  file  for  reference. 

Participants  were  told  that  up  to  one  hour  was  allotted 
for  each  oroblem,  and  that  the  computer  would  automatically 
stop  the  problem  at  that  time.  Participants  were  permitted  ‘  o 
take  a  short  break  between  test  problems  if  they  desired. 

'Jata  Entry.  Participants  used  the  key  pad  to  enter  data 
into  the  computer.  A  oamicipant  could  select  from  the  following 
function:  ■ . 
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Function  "  '  :  Chain!''  'he  1  -'ape  ul  Lx.^m!'1'-1 
Solution  .  '  o  ciV'i'V  lae  ox  an  >  c  solution  ;  '.  e 

on  the  terminal  display ,  a  uartie r'.'i'.  would  nr.  >, 

Key  #1  follow  a  by  1  le  L..V  ’  1_K  '  -y.  The  comp'-  -  n 
would  then  request  the  ratio  •  .  mber  that  t^e  ^amticipent 
wanted  to  view.  '1  '  a  part  pant  we  Id  enter  _  e 

Daae  number,  ••  ult’.nq  .n  a  d:.--  .._.y  Of  the  desired 

pat!'  A  p.iae  is  a  term  r.  •^rinq  to  Cat.'  iiforma-.oi  ■ 
trial  can  be  presented  on  a  v  •<  >  >»u*">‘.i'r  at  a  qiv  an 
time .  Add’d  lone,  tvv'os  are  usee  .o  present  additional 

informal  ion.  'i'  t:>i-.  case  '->e  rnpes  cor*.  ■  - :  too 


sou  i  oar  ■ :  j'.r  ■ 


ac- 1  examp -e 


,-tM.ild.  disnlay  ■"  ih!  xai"'.',  .n\-':o-v  a, O'  i  as  each 

P«  s  It'  1  ’•  e'  f  1  '•  •  '  ’  ,  ir  K  ’’  H  ■  \  me  w  v.  <  '  d  no  ns  -  ,  tenet* . 

Function  ti  ':  nler  Kxa.niple  -olu'’  .  W:'t'n 

the  i ’  n’.  ie  tra’  v.  wc  ed  to  e''ter  an  ex. imp''  ■  .o'.w  '  :  ■  , 

•tt‘/‘.  ie  ( ire;  •  ed  "  -  and  l  •  'e* '  'he  1_N  '  FK  Fey. 

I  he  con  oc.er  t;>e-  ..  'e-  ,je  •  'a  >e  !i.!”''  •'  1 s  'to-, 

in  'he  .  •  i'mpV  .  'die  oartic  na.nt  wnu'e’  ■  -id  w:t;> 


•  ie  anproprinte  number  and  ‘  !'e"  nn  e.r  1 
e  1 '  '  nun  -ber  - . 


i  «•  to-'  wp.  C-hanne  ata- .  ot  L.xamp’..  taolu'. too. 

1  n  eh,  v  ure  the  .re  o'  any  on  l At.  ..  —  ■'  t  •  A!\(_F 

tee  .  ..  e  ■-  .V'le'ti  nre  >  •  'Fey  '  '  ’o' lowed  bv  ie 
1  \  TL't  Key .  '  ’e  coimuler  would  n  reuue:  •*.  ‘e'e 

number  o!  'pc  exuninlo  •  atio*'  'no  narticinant  wis-iet! 

•  o  e;v .  1  'res-  -  too  Key  -Ft  c:ianoed  ‘e'e  statue  >r  the 

examn'i  .ole' ion  !"(  Ai  X  .1  ’’  to  CbAW  ...  L.  on  from 

CAVCL.  to  ACCf.F7  .  A. .  eha'i  we’-.'  o.utere  F  the 
eonintiter  end.  '  ■  '  -'nt>  d  v  to  con' -a**"  to  t!'e  new 

Ait"!  ’  '  !  /'  "AN  .'ale  .  t'n'v  act  ooied  >  ■x.i"i,,!c 

.1 1  ‘  c  1 1 ;  •  '  worn  •  -1  1  IP  lit  let  :  :  III  !• !  the  v  am".  < 

•  .l.iti  •  nor'  1  ■  a  •'  h.  A  •.  aneet 'ed  exa'ii'li1 

,  .  v,  •  ..'ed  --y  •'  -e  ."'i!'Ut-  r.  Any 


a  d  1  v  ’. 1  O'  •  '.hi  -  nr-  .  ed 

.  ,  '  •  •  V  >’  •  1  V  C\  If  'C  l.  •  ’  !  L'C 


•  t  ’•  •  *1  n  I  ( 
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Function  Commanc  .'ne  Computer  ‘.o  Generate 
Solutions .  As  a  participant  entered  sol.  '.ons,  the  computer 
generalized  them  to  form  a  ship  selection  logic  (SSL). 

To  be  sure  that  this  generalization  was  correct,  a 
participant  could  command  the  computer  'o  generate  and 
display  its  own  examole  solutions.  Thi-  was  done  by 
presssing  Key  #4  fol.owed  by  the  ENTER  Ley.  After 
viewing  the  computer's  example  solution,  a  participant 
could  either  accept  or  cancel  .  ~ro  accept  a  solution,  the 
participant  pressed  Tc  ENTER  Ley;  to  cancel  a  solu¬ 
tion,  he/she  pressed  Key  "  followed  by  the  ENTER  Ley. 

Function  #5:  View  the  Computer's  SSL.  T o 
view  the  SSL  formed  by  'm  computer  (bcu.ed  on  the 
example  solutions  inuut  by  the  -participant)  the  oarti- 
cipant  pressed  Key  -‘5  r-'oltowod  by  the  EN^ER  <ey. 

Upon  such  a  rep1-  the  computer  would  display  the 

various  ship  combination  '■  that  together  T  -med  be 
computer's  SSL. 


Data  Collect-' -m 


All  example  elutions  were  entered  by  participants 
into  the  computer.  The  computer  t  _n  recorded  all  entmes, 
as  well  as  the  time  expected  for  each  problem  solution. 

This  method  of  collecting  data  enab  generation  of  a  printout 
listing  detailed  information  on  each  oartic ’pant's  activities 
during  the  experiment. 

Performance  Measures 

Probability  of  Correctly  Selecting  an  Acceptable  S:~-; p. 
Two  summary  performance  measures  were  developed  to  re¬ 
veal  how  well  each  participant  solved  the  test  problems.  The 
two  measures  were  identical  with  respect  to  scoring  correct 
ship  combinations,  but  differed  in  that  Derformr  -ce  Measure 
2  .rovided  a  po'-'-’ty  for  an  incorrect  ship  com';  '"ion. 

"he  measures  wc-e  constructed  by  scoring  the  MIN/ 
MAX  transiting  and  static^  timers  for  each  ship  type.  As 
shown  in  Figure  2,  the  range  of  correct  values  for  the  M'N/ 
MAX  transiting  and  stationing  times,  which  is  a  function  of 
cne  ships  on  ‘he  available  ship  list,  can  be  viewed  as  r  area 
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(ACS)  Area  of  correct  solution  =  (MAXTC  -  MINTC)  (MAXSC  -  MINSC) 
(APS)  Area  o€  oart'.c inarv.' s  solution  =  (MAXTP  -  MINTP)  (MAXSP  - 
.V.INSP) 

MINSP 
MINSC 
MINTC 
MINTP 
MAXSC 
MAXSP 
MAXTP 
MAXTC 


Minimum  Stationing  -  Participant 
Minimum  ^Stationing  -  Correct  Solution 
V  mi  mum  transiting  -  Correct  Solution 
Minimum  Transiting  -  Participant 
Maximum  Stationing  -  Correct  Solution 
Maximum  Stationing  -  participant 
Maximum  Trans:*-ing  -  Participant 
M»ixm  -on  ""ransiting  -  Correct  Solution 


Figure  2.  Method  of  Computing  Probability 
of  Acceptal  *  t  Ship  Selection 
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in  space.  Every  ship  of  a  particular  type  whose  coordinates 
fall  within  that  area  is  acceptable.  However,  if  a  particioant's 
ship  selection  logic  provides  a  minimum  for  eit  er  transiting 
or  stationing  lime  greater  than  the  correct  minimum,  ^r  if 
the  maximum  for  either  transiting  o^  stationing  is  less  than 
the  correct  maximum,  then  the  "area"  specified  by  the  parti¬ 
cipant  will  be  container  entirely  within  the  correct  "area." 

As  a  result,  it  is  possible  that  a  ship  with  acceotable  coordinates 
might  not  be  selected  according  to  the  SSL  specified  by  a  giv  ■> 
participant.  The  probability  that  an  elicit  e  ship  of  given 
type  (denoted  for  convergence  by  the  ■  adscript  i)  will  be  selected 
by  the  participant’s  SSL  is  the  ratio  of  the  areas  just  c' .'.scribed. 
That  is: 

APS 

Pi  “  ACS  (7) 

where:  APS  =  Area  of  participant's  SSL  for  shio  tyoe  i. 

ACS  =  Area  of  correct  SSL  for  ship  type  i. 

The  probability  of  selecting  all  eligible  ships  for 
a  correctly  specified  combination  of  ships  is  tne  product 
of  the  P.  over  all  the  ship  types  in  that  combination.  That 
is: 


PC  -7 TP  over  all  i  in  combination  k  (8) 

k  i 

where:  K  represents  the  kttn  correctly  specified  snip  combination. 

If  a  problem  requires  N  correct  ship  combinations,  the  average 
probability  of  selecting  an  eligible  ship  regardless  of  type, 
which  we  shall  call  Performance  Measure  1  ,  or  PM1  ,  is: 


N 

PM  1  ~  f"j  2  PC  (9) 

k-1 

Performance  Measure  1  reflects  the  orobability  that  acceptable 
ships  are  in  fact  accepted  by  the  SSL  specified  by  t’"?  parti¬ 
cipant.  If  a  particular  ship  combination  is  not  specified  by 
the  participant,  the  ^-mesponding  PC  value  is  zero.  Note 
that  PM  l  does  not  carry  any  penalty  For  specifying  incorrect 
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ship  combinations.  A  measure  that  does  penalize  for  incorrect 
combinations,  PM2,  is  given  as  follows: 

1  N 

PM2  -  -  [(  S  PC,  )  -  n  (10) 

N  k=1 

where:  l  is  the  number  of  incorrect  ship  combinations. 

PM1  and  PM2  were  the  two  summary  measures  us  ' 
to  analyze  participants'  cat  a  . 

Procedure  Measure 

Two  participant  strateoy  measures  were  developed  to 
test  the  relationship  between  the  strategy  used  by  oarticipants 
in  developing  example  solutions  and  their  resulting  performance 
scores.  One  strategy  measure,  termed  a  procedure  measure, 
was  des;gned  to  examine  the  pattern  cp  choices  among  the  options 
available  to  participants  for  working  with  the  computer. 

Referring  to  Figure  3,  it  can  be  seen  that,  once  a  Darticipant 
entered  an  example  sc  ution  into  the  stouter,  h  .  /she  had 
three  choices  for  the  next  step.  One  c  oice  was  to  input  another 
example  solution.  Another  choice  was  to  review  the  computer 
ship  selection  logic  (SSL)  to  determine  the  er  ct  of  all  example 
solutions  entered  up  to  the  present  time.  The  third  choice 
was  to  request  that  the  computer  generate  its  own  example 
solutions .  The  relative  frequency  of  these  three  choices  is 
represented  by  the  Probabilities  P  ,  P  ,  and  P  ,  respectively. 

(  c-  O 

The  value  of  P^  reflects  the  propensity  of  the  particiDant 
to  input  either  a  single  example  solution  or  multiple  example 
solutions  in  sequence.  If  a  participant's  P  has  a  low  value  - 
near  0  -  it  may  be  concluded  that  the  participant  tends  primarily 
to  input  single  example  solutions.  The  participant  then  either 
requests  a  display  of  the  SSL  or  requests  that  the  computer 
generate  example  solutions.  If  P  ;s  very  high  -  rear  i  .0  - 
then  the  value  indicates  that  multiple  examples  were  input  before 
proceeding  to  c  of  the  other  input  states.  In  fact,  the  value 
of  P  provides  an  estimate  of  the  mean  number  or  times  example 
solutions  are  input  lry  seauence,  as  follows: 

Mean  number  of  entries  in  sequence  =  (11) 

1  - 

1 
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Figure  3.  Par.Lcipant  Procedural  Strategy 


In  a  similar  way  the  values  of  P2  and  Pg  reflect  the 
propensity  of  a  particioant  to  view  the  SSL  or  to  request  the 
generation  of  sample  solutions,  respectively.  A  large  value 
of  P  ,  for  instance,  indicates  a  strategy  of  inputting  a  single 
(or  only  a  few)  example  solutions  and  then  reviewing  the  effect 
on  the  SSL.  A  large  value  of  P^  inaicates  a  propensity  to 
input  one  (or  only  a  few)  input  examples  and  then  to  request 
the  computer  to  generate  example  solutions. 

Since  the  three  alternatives  represent  all  of  the  avail¬ 
able  choices,  the  sum  P  +  P^  +  P  will  always  equal  1  .0. 

Thus,  as  P,  increases,  Ihe  sum  P  +  P  will  automatically  de- 
1  2  3 

crease.  As  a  result,  the  three  variables  are  correlated,  and 
any  two  may  be  used  in  a  regression  as  independent  variables 
to  investigate  the  relationship  between  participant  strategy  and 
L.sk  performance. 

Combinational  Measure  (CM) 

Another  mease  -c  of  oarticipant  strategy  involves  the 
nature  of  the  relation  shin  between  one  example  solution  and  the  previous 
solutions  input  by  a  participant .  In  each  problem  logical  OR 
conditions  are  specified  which  force  a  participant  to  develop 
several  different  shin  ombinations  in  order  to  completely 
specify  an  SSL.  Many  of  these  combine.*- 'ons  require  variations 
within  one  ship  type  -  submarine,  for  example,  either  nuclear 
(SSN)  or  non-nuclear  (SS)  -  while  others  may  require  variations 
over  more  than  one  tyoe. 

If  two  submarines  are  specified  in  conjunction  with 
other  ship  types,  then  each  specific  con  nation  of  the  other 
shios  must  be  associated  with  three  possible  variations  of 
SSN's  and  SS’s,  as  follows: 

2  SS  and  (other  ships) 


1  SS  and  1  SSN  and  (other  ships) 


SSN  and  (  -ther  ships) 
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Similarly,  if  3  submarines  are  specified,  then  each  specific 
combination  of  other  ships  must  be  associated  with  four  varia¬ 
tions  of  SS's  and  SSN's  as  follows: 

3  SS  and  (other  ships) 


OR 


2  SS  and  1  SSR  and  (other  ships) 


OR 


1  SS  and  2  SSN  and  (other  ships) 


OR 


3  SSN  and  (other  ships). 

Often,  variations  (i.e.,  "OR"  conditions)  within  one  ship  type 
(e.g. ,  submarine)  will  be  specified  in  combination  with  variations 
(OR  conditions)  within  one  or  more  of  another  type  of  ship  (e.g., 
nuclear  or  non-nuclear  aircraft  carrier).  Variations  over 
multiple  ship  types  require  the  generation  of  numerous  combina¬ 
tions,  and  it  is  difficult  to  generate  all  such  combinations  in 
the  absence  of  a  systematic  method . 

The  combinational  strategy  measure  is  designed  to  detect 
whether  a  participant  tenc  .  to  construct  example  solutions 
systematically  by  effecting  changes  only  within  one  ship  type 
(e.g.,  SS  or  SSN)  in  successive  example  solutions  and  by 
providing  all  such  required  variations  within  one  ship  type  in 
sequence  before  proceeding  to  the  next  type. 

The  measure  is  computed  as  follows: 

1  .  If  on  two  successive  solution  examples  more  than 
one  ship  type  is  changed  -  score  0. 

2.  If  on  two  successive  solution  examples  changes 
occur  only  within  one  ship  type  -  score  1  . 

3.  If  on  three  successive  solution  examples  changes 
occur  only  within  one  ship  type  -  score  1  for  the 
first  change  (according  to  condition  2  above)  and 
2  for  the  second  change . 


45 


4.  In  general,  for  multiple  successive  solution 
examples  where  changes  are  made  only  within 
one  ship  type  -  score  1  for  the  first  change, 
score  2  for  the  second  change,  score  3  for 
the  third  change,  etc. 

5.  The  total  score  is  the  sum  of  the  individual 
scores  divided  by  V'--  maximum  score  possible 
for  the  problem. 

This  measure  is  termed  "combinational , "  since  the  extent  to 
which  ship  combinations  are  generated  in  a  systematic  manner 
is  the  area  of  interest.  A  low  value  of  the  measure  reflects 
the  frequent  use  of  input  examples  in  which  changes  are  made 
successively  over  more  than  one  ship  type. 


DATA  ANALYSIS 

The  term  "treatment"  in  the  paragraphs  below  refers 
to  the  effect  on  the  performance  data  of  variations  in  either 
problem  or  processor  complexity .  Two  treatments  may  thus 
be  applied,  either  individually  or  in  co~  bination,  to  the  analysis 
of  the  performance  of  participant  categories  • 

The  following  analyses  wnre  conducted: 

Analysis  1 

An  ANOVA  was  used  to  de*:';'cnnine  the  significance  of 
the  effect  on  performance  of  the  two  participant  r  ateaories  and 
the  two  treatments.  Two  ANOVAs  were  -  undue  ted,  one  each, 
for  Performance  Measures  1  and  2. 

Analysis  2 


An  aposteriori  multiple  comparison  of  means  test 
(Student-Newman-Keuls  [SNK])  -  as  used  to  rmine  whether 
the  means  for  Performance  Measure  1  for  each  level  of  the 
two  treatments  per  participant  category  were  significantly  different. 

In  addition,  the  SNK  test  was  applied  to  the  performance  means 
for  the  nine  treatment  combination  cells  for  each  participant  category. 
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Analysis  3 


A  performance  count  was  conducted  to  determine  the 
number  of  participants  that  achieved  perfect  scores  in  each 
treatment  combination  cell. 

Analysis  4 

Two-tailed  t-tests  were  used  to  determine  whether  the 
performance  scores  achieved  by  participants  assigned  to  various 
participant  categories  differed  significantly. 

Analysis  5 


A  univariate  linear  regression  analysis  for 
five  independent  demographic  and  two  in<.  oendent  treatment 
variables  was  conducted.  The  variables  were,  in  the  order 


mentioned: 

1  . 

Participant 

group 

2 . 

Participant 

pre-test  score 

3. 

Participant 

age 

4. 

Part.~-.pant 

years  of  professional  experience 

5. 

Participant 

years  of  higher  education 

6. 

Problem  complexity  level 

7  . 

Processor 

complexity  level 

The  dependent  variable  was  the  score  the  participants  achieved 
on  each  test  problem  using  Performance  Measure  1  .  Also,  a  Step- 
Wise  Linear  Regression  was  conducted  using  the  seven  variables. 

Analysis  r 

To  the  variables  in  the  analyses  above  were  added  the 
three  independent  strategy  measures,  ,  P  ,  and  Combinational 
Measure  (CM).  (Note  that  P  is  not  an  independent  variable  since 
p^  +  P^  +  Pg  =  1).  The  complete  list  of  all  ten  independent 
variables  therefore  concluded  with: 
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Procedural  Strategies 


10.  Combinational  Measure  (CM) 

Univariate  linear  regression  results  were  obtained  for  these 
additional  variables,  and  a  second  Step-Wise  Linear  Regression, 
over  all  ten  of  the  variables,  was  conducted. 

RLSULTS 

Summarized  Performance  Data 

Mean  performance  scores  using  Performance  Measure  1 
(probability  of  selection  of  an  acceptable  ship)  for  each  experiment 
cell  (i.e.,  each  combination  of  the  treatments)  and  each  participant 
outegory  are  given  in  Table  9.  Similar  data  were  also  obtained 
using  Performance  Measure  2  (probability  of  selection  of  acceptable 
ship  minus  a  penalty  for  selecting  an  unacceptable  ship). 

These  data  are  given  in  Table  10. 

Performance  Measure  1  results  achieved  by  programmers 
show  ■>  decrease  in  performance  with  increasing  levels  of  problem 
and  processor  complexity.  Performance  Measure  1  scores 
achieved  by  bookkeepers/accountanU  are  lower  in  every  cell 
than  those  of  the  programmers,  but  do  not  reveal  a  monotonic 
decrease  with  increasing  problem  and  processor  complexity.  A l  o, 
the  participants  in  group  5  (bookkeepers/accountants)  (re:  erence 
Table  3)  "appear"  to  have  higher  scores  in  cells  A  B  , 

than  would  be  expected  if  the  three  bookkeeper/accov'-  baht 
groups  (4,  5,  6)  had  included  individuals  of  equal  caoability. 

This  result  is  further  evaluated  in  Analysis  4. 

Comparison  of  the  performance  scores  obtained  with 
Performance  Measure  1  and  Performance  Measure  2  (Tables 
9  and  10)  >-  veals  that  performance  evaluated  with  and  without  a 
penalty  for  errors  of  commission  differs  only  slighb’y  and  shows 
the  same  trends  in  effects  for  both  participant  categories,  ->"d 
for  both  problem  and  orocessor  complexity  levels .  The  reason 
for  this  is  that  only  a  few  errors  of  commission  went  undetected 
(and  therefore  not  corrected)  by  the  participants .  An  error  of 
commission  results  when  the  SSL  specifies  one  or  more  ships 
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jblem  complexity  levels(  in  increasing  order  of  complexity) 

)ces:,or  complexity  levels  (B  having  the  most  jeneralization  and 
having  the  least  generalization 


that  are  unacceptable  according  to  the  problem  instructions. 

The  performance  scores  chat  result  from  Performance  Measure 
1  are  a  more  direct  measure  of  the  effects  of  errors  of  omission  - 
a  more  serious  problem. 

Plots  of  mean  scores  using  PM1  for  the  three  experiment 
problems  versus  the  pre-test  sco’e  for  each  participant  category 
are  shown  in  Figure  4,  and  plots  of  mean  performance  scores 
(PM1 )  ^r  each  participant  category  versus  problem  complexity 
are  shown  in  Figure  5.  A  similar  plot  for  processor  complexity 
from  the  user's  viewpoint  is  given  in  Figure  2.  These  data 
suggest  that  there  is  a  positive  correlation  between  scores  on 
the  experiment  problem  and  pre-test  score,  and  that  increasing 
problem  complexity  results  in  an  initial  reduction  of  performance 
followed  by  a  leveling  or4'  at  higher  complexity  levels.  In 
contrast,  the  effect  of  increasing  processor  comolexity  is  an 
apparent  monotonic  decrease  in  performance. 

Analysis  ’  .  ANOVA 

ANOVA  results  for  the  probability  of  selectim  an  accept¬ 
able  sh' o  (Performance  Measure  1)  are  shown  in  Table  11.  In 
this  analysis,  the  main  effects  of  participant  category,  problem 
complexity,  and  processor  complexity  were  all  found  to  be  sig¬ 
nificant:  F  (1,  54)  =  21.38,  p  <  .001;  F  (2,  108)  =  7.53,  p  ^ 

.005;  and  F  (2,  108;  =  35.02,  p .001,  respectively. 

ANOVA  results  for  the  probability  of  selecting  an  accept¬ 
able  ship  minus  the  penalty  for  selecting  an  unacceptable  ship 
(Performance  Measure  2)  are  shown  in  Table  12.  Again,  sig¬ 
nificant  main  effects  fo-  participant  category,  problem  complexity, 
and  processor  compk  ty  were  found:  F  (1  ,  54)  =  21.96,  p 
.001;  F  (2,  108)  =  8.34,  £  ^  .005;  and  F  (2,  108)  =  30.32;  p  ^ 
.001,  respectively.  No  significant  interaction  ejects  were  found 
in  either  analysis. 

The  probable  reason  that  similar  results  were  obtained 
in  the  two  analyses  is  that  very  few  errors  of  commission  were 
made.  Thus,  the  measure  that  penalizes  for  such  errors  (PM2) 
provides  results  that  are  essentially  the  same  as  those  that 
are  provided  by  the  measure  that  ignores  such  errors  (PM1 ). 
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Mean  I  Ml  Scores  for  the  Three  Experiment  Problems 
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^igure  4.  Mean  PM1  Scores  on  Three  Experiment  Problems 

Versus  '  -Test  Scores 
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Figure  5.  Mean  Scores  (PMi)  for  Each  Participant  Category 

Versus  Problem  Complexity 
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Figure  6.  Mean  Scores  (PM1)  for  Each  Participant  Category 
Versus  Processor  Complexity  (User's  Viewpoint) 


Table  1  1 


ANOVA  Results  her  Performance  Measure  1 


Source  of  Variation 

SS 

DK 

MS 

F 

Between  Subjects 

1 7 . 422 

59 

C  (Participants) 

4 . 755 

1 

4.755 

21 .332** 

Rows 

.252 

o 

.  '20 

.567 

C  X  Rows 

.404 

o 

.202 

.910 

Sub  Within  Group 

12.010 

54 

.222 

Within  Subjects 

0.24  i 

120 

I  A  -  Problem 

.  two 

o 

.  3 1  5 

7.537* 

!  Complexity 

B  -  Processor 

2.950 

o 

1  . 405 

35.024** 

Complexity 

AC 

.  005 

o 

.032 

.787 

BC 

.  0 1  0 

o 

.008 

.  194 

AB 

.078 

2 

.039 

.940 

ABC 

.000 

2 

.000 

.009 

Error  (Within) 

4.518 

108 

.041 

*  p  <  .005 

**  p  .001 

Ref:  Plan  #9,  pgs .  727 

-  730,  Static 

tier’ 

Principles 

in  Experimental 

Design,  Winer 

1971). 
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Table  12 


ANOVA  Results,  tor  Performance  Measure  2 


ss 

DF 

MS 

F 

Source  of  Variation 

Between  Subjects 

17.389 

59 

C  (Participants) 

4.834 

1 

4.834 

2  ’  . 905 : 

Rows 

.243 

2 

.121 

.  553 

C  X  Rows 

.425 

2 

.212 

.  966 

Sub  Within  Group 

1 1 .885 

54 

.220 

Within  Subjects 

8.642 

120 

A  -  Problem 

.763 

2 

.38’ 

8.344 

Complexity 

B  -  Processor 

2.776 

* ' 

’  '88 

80.328 

Complexity 

AC 

.061 

o 

.  330 

.  668 

BC 

.02  1 

o 

.  0  ’  0 

.  230 

AB 

.056 

2 

.  028 

.614 

ABC 

.020 

o 

.010 

.  .  2  ’  9 

Error  (Within) 

4.943 

1  08 

.0-0 

*  p  ^  .005 

**  p  ^  .001 


Ref:  Plan  #0,  pgs.  727  730,  Statistical  Principles  m  1  xpcrimont 

Design,  Wine*-'  (1971). 
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Analysis  2:  Aposteriori  Test: 
CornDQrison  of  Treatment  Means 


Tables  13  and  14  show  the  results  of  Studcnt-Newman- 
Keuls  (S  MK)  tests  (Sokal  &  RohT,  1969)  on  the  mean  scores 
across  all  levels  of  problem  complexity  for  programme  and 
bookkeepers/accountants,  respectively.  Tables  15  and  16  show 
results  of  the  SNK  tests  on  the  mtoi  scores  across  all  levels 
of  processor  complexity  for  programmers  and  bookkeepers/ 
accountants,  respectively.  Tables  17  and  18  show  Sk'K  results 
for  f.o-way  comparisons  of  the  mean  scores  for  ihe  programmer 
and  bookkeeper/accountant  experiment  cells,  respectively. 

Results  shown  in  Tables  13  an<_  :■  indicate  boat  problem 
complexity  affected  the  performance  of  programmers  significantly, 
but  that  processor  complexity  did  not  have  a  significant  effect. 

In  contrast,  the  results  shown  in  Tables  14  and  16  suggest  that 
problem  complexity  did  not  effect  the  performance  of  bookkeepers/ 
accountants  significantly,  but  that  processor  complexity  did  sig¬ 
nificantly  affect  their  performance.  Table  17,  which  is  a  more 
detailed  analysis  dealing  with  individual  treatment  cells  (i.e., 
combinations  of  processor  and  problem  complexity  levels), 
indicates  that  the  performance  of  programmers  was  significantly 
better  with  Processor  Level  B  at  all  problem  levels  than  with 

Processor  Level  B,„  at  Problem  Levels  A  and  A  .  With  Processor 
3  o  3 

Level  B^,  significantly  better  performance  is  achieved  only  at 

Problem  Levels  A^  and  A^ .  These  results  suggest  two  principles 

of  problem/processor  usage: 

1.  A  high  level  of  processor  generalization  (i.e., 
low  complexity  from  the  ser’s  viewpoint)  aids 

a  computer  user  in  accurately  specifying  problem 
solutions . 

2.  As  problems  become  more  complex,  there  must  be 
a  corresponding  increase  in  processor  generaliza¬ 
tion  capability  to  support  superior  performance. 

It  is  apparently  important  for  the  experienced  computer  user  to 
be  able  to  select  processor  generalization  capability  as  a  function 
of  the  complexity  of  the  problem  solution  to  be  specified. 
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Table  13 


Studeri  -  Vewman  -  Keuls  1  csl 
Results  For 


Problem  Complexity-Programmers 


Problem 

Means  * 

A  __ 

.504 

A 

2 

.520 

A 

1 

.67  G 

A3  . 504 

!  i 

:  .  ooo 

i 

1 

j 

j 

1 

i 

1 

A_  .  520 

2 

;  i 

|  -016  ; 

.000 

1 

|  *  *  , 

♦  * 

A1  .676 

.172 

i  i 

1  j 

I 

i 

.  1  56 

.000 

*Mean  of  all  programmers'  tierformances  on  each  problem,  i.e., 
A1  ’  A<j’  A3’  using  PM  •  • 

A^  j  A0,  A  are  the  problem  complexity  levels  (in  increasing  order 
of  complexity) 

Each  mat  'ix  entry  =  row  vo’  n  -  column  value  (e.g,  ,  entry  ir  the 
third  row,  first  column  is  .172  =  .676  -  .504) 

** Indicates  row  value  is  significantly  different  from  column  value 
£  ^'.05. 
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Table  14 


Student  -  Newman  -  Keuls  Test 
Results  For 

Problem  Complex  -  Ly-Bookkeepers/Accountants 

Problem  A  A  A 

3  3  1 

Means*  .205  .220  .300 


Aq  .205 

- , - 

1 

j 

.  000  | 

1 

i 

t 

- 

n/s  1 

j 

A  .220 

3 

•015  i  .000 

» 

n/s  a/s 

A  i  . 300 

.095  !  .000 

1 

; 

.000 

*Mean  of  a11  bookkeepers  '/accountants'  performances  on  each 
problem,  i.e.,  A1  ,  Ar>,  A,^,  using  PM1  . 

A^  ,  A  ,  A^  are  the  problem  complexity  levels  (in  increasing  order 
of  complexity) 

Each  matrix  entry  =  row  va’c  column  vale  -  (e.g. ,  entry  in  the 
third  row,  first  column  is  .095  =  .300  -  .205 

n/s  =  Not  significant 
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Table  15 


Student  -  Newman-Keuls  Test 
Results  For 

Processor  Complexity- Programmers 


Processor 

Level 

B1 

B3 

B2 

Means  * 

.544 

.  56 1 

.  595 

B  .544 

1 

.000 

, 

i 

:  i 

i 

' 

n/s 

B  .561 

3 

.017 

.000  ! 

i 

i 

i 

i 

n/s 

|  n/s 

|  i 

| 

B„  .  595 

2 

.051 

.  034 

i 

1 

1  . 000  I 

i  1 

i  ! 

‘Mean  of  all  programmers'  performances  on  each  processor  level 
B1*  B2’  B3’  using  PM1  • 

B1  ,  B  ,  R  are  the  processor  complexity  levels  (B^  having  the  most 
generalization  and  B  having  the  least  generalization) 

O 

Each  matrix  entry  =  row  value  -  column  value  (o.g. ,  entry  in  the 
third  row,  first  column  is  .051  =  .595  -  .544) 

n/s  =  Not  significant 
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Table  16 


Student  -  Newman  -  Keuls  Test 
Resu’ts  For 

Processor  Complexity-Bookkeepers/Accountants 


Processor 

B 

B 

Level 

3 

2 

1 

Means* 

.151 

.256 

.351 

B3  .151 

1  ' 

j  ! 

.  000  ! 

i 

i  i 

*  * 

B2  .256 

.105 

.000 

1 

*  * 

*  ♦ 

1 

B  .  35  1 

.200 

.  O'  1  ■ 

.  000 

‘Mean  of  all  bookkeepers'/accountants*  performances  on  each 
processor  level,  i.e.,  L-  ,  ,  B^,  B^,  using  P/Vi  t 

,  BQ,  By  are  the  processor  complexity  levels  (R  having  the 
most  generalization  and  B  having  the  least  generalization) 

Each  matrix  entry  =  row  value  -  column  value  (e.g.  ,  entry  in  the 
third  row,  first  column  is  .200  =  .351  -  .151 

*  *  Indicates  row  value  is  significantly  different  from  column  value  p 
*2.05. 
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Data  presented  in  Table  18  indicates  that  performance 
with  Processor  Level  Bg  for  all  problem  levels  for  book¬ 
keepers/accountants  is  significantly  lower  than  for  problem/ 
processor  complexity  combinations  and  AgB^  .  Table  18 

also  contains  other  treatment  combinaitions  that  provide  statistically 
significant  differences  in  performance,  such  as  cell  (treatment 
combination)  A^B^  which  leads  to  significantly  improved  per¬ 
formance  over  that  of  cells  A  Bg  and  A  B  .  These  results  are 
more  difficult  to  summarize  than  those  for  programmers; 
however,  they  support  the  suggested  principles  of  problem/ 
processor  usage  giver  above. 

Analysis  3:  Percentage  f'p  Participants 
_ Achieving  a  Perfect  Score _ 

Table  19  shows  the  percentage  of  participants  (both 
programmers  and  bookkeepers/ accountants)  who  achieved  a 
perfect  score  for  each  experimer  treatment  combination.  This 
table  indicates  that  on  Problem/Processor  Combination  A^B^  60% 
of  the  programmers  achieved  a  perfect  score,  with  a  lower 
percentage  achieving  perfect  scores  in  other  cells .  It  was 
possible  for  participants  to  achieve  a  perfect  score  on  any  of 
the  problem/processor  combinations  presented. 

Analysis  4:  t-Test  of  Group  Mean  Scores 

Table  20  shows  the  results  for  t-tests  performed  to  de¬ 
termine  whether  the  mean  scores  for  the  three  groups  in  each 
participant  population  were  different.  No  evidence  was  found  to 
reject  the  null  hypothesis  that  the  participant  groups  did  not 
differ . 

Analysis  5:  Step-Wise  Linear  Regression  Analysis  #1 

Each  of  the  seven  variables  was  tested  in  a  univariate 
regression  with  results  shown  in  Table  21  .  Among  the  seven 
variables,  pre— test  score  was  the  best  univariate  predictor  of 
performance,  suggesting  that  initial  participant  capability  had 
a  strong  influence  on  performance  scores.  Next  in  importance 
in  score  prediction,  as  determined  by  the  amount  variance 
explained  in  a  univariate  regression,  were:  participant  cateogry, 
participant  age,  processor  complexity,  and  participant  years 
of  experience,  in  that  order,  followed  by  years  of  higher 
education  and  problem  complexity. 
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Tabte  20 


t-Test  for  Differences  in 

Participant  Groups 

Programmer  Groups 

DF 

t-Score 

Group  1  compared  with  Group 

2 

18 

.  1980 

n/s 

Group  1  compared  with  Group 

3 

18 

.2020 

n/s 

Group  2  compared  with  Group 

3 

18 

.0227 

n/s 

Bookkeeper/Accountant  Groups 

DF 

t-Score 

Group  4  compared  with  Group 

5 

18 

1 . 4209 

n/s 

Group  4  compared  with  Group 

6 

18 

.6152 

n/s 

Group  5  compared  with  Group 

6 

18 

1 . 1 960 

n/s 

Note:  n/s  =  Not  significant 

Two-tailed  t-score  required  tor  significance  at  the 

.05  level  -  2.101 

Group  1  and  Group  4  used  problems  15/2,  52/3,  31/1 
assigned  m  random  order. 

Group  2  and  Group  5  used  problems  15/1,  52/2,  31/3 
assigned  in  random  order. 

Group  3  and  Group  6  used  problems  15/3,  52/1,  31/2 
assigned  in  random  order. 


fell] 


Pearsorvs-R 

Correlation  with 

the  Dependent  Variance 


Variable 

Exolained 

F-Ratio* 

Pre-test  Score 

.674 

46% 

1 43 . 382 

Participant  Category 

.437 

19% 

40.515 

Participant  Age 

-.359 

13% 

25.423 

Processor  Complexity 

-.335 

1  1% 

21 .681 

Participant  Years 
Experience 

-.206 

4% 

7.657 

Participant  Years 

Higher  Education 

.196 

4% 

6.889 

Problem  Complexity 

-.156 

2% 

4.289 

Combinational 

Measures  (CM) 

.796 

63% 

296.542 

* 

CM 

Q. 

-.527 

28% 

66. 125 

P1* 

.384 

15% 

29.752 

*  Procedural  Measures  j 

see  p. 

42. 

**All  scores  are  significant  a',  the  <  .001  level 
F  . 001  [5,1 68]  =  4.10. 
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Table  2 2  provides  the  results  of  a  Steo-Wise  Linear 
Regression  analysis  using  performance  scores  un  PM1  as  the 
dependent  variable  and  seven  candidate  independent  variables. 

The  seven  independent  variables  tested  in  the  Step-Wise  Linear 
Regression  were:  pre-test  score,  processor  complexity, 
participant  category,  problem  complexity,  participant  years  of 
experience,  participant  years  of  higher  education,  and  participant 
age.  The  first  five  of  these  accounted  for  65%  o'  .he  variance; 
the  addition  of  the  remaining  two  incr<_u_.ed  the  amount  of  variance 
explained  by  less  than  1%. 

Table  23  shows  a  regression  table  for  the  irst  five  of  the 
independent  variables  listed  above.  The  addition  of  the  remaining 
variables,  participant  age  anu  higher  education  were  found  not  to 
increase  the  variance  explained.  In  this  table  it  can  be  seen  that  pre¬ 
test  scores  correlate  very  highly  with  the  dependent  variable  experiment 
scores  (.674).  Also,  participant  category  correlates  hiyhly  with 
the  dependent  variable  experiment  scores  (.437).  In  the  uni¬ 
variate  regression,  the  coefficients  for  pre-test  score  were  found 
to  be  highly  significant  (t_  (173)  =  .  u  24,  p-  .o01),  as  were 

those  for  participant  group  (t_  (173)  =  4.663,  p  --'.001)  and 
processor  complexity  (t_  (173)  =-7.367,  p  ^  .001).  Also  found 
to  be  significant  were  the  coefficients  for  orob’.om  complexity  (t_ 

(173)  =-3.175,  p  ^t.01)  and  participant  experience  (t_  (173)  = 

-3.604,  p  ^.001).  The  regression  equation  was  also  ...uruficant 
(F  (5,16b)  =  61.511,  p  .001). 

Analysis  6:  Step-Wise  Linear  Regression  Analysts  #2 

Table  24  provides  the  results  of  a  second  Step-Wise 
Linear  Regression  analysis  using  scores  on  PM1  as  the  de¬ 
pendent  variable  and  the  following  independent  var'ables: 
procedure  measures  and  Pjf  combinational  measure  (CM), 
pre-test  score,  age,  experience,  education,  problem  complexity 
and  processor  complexity.  Of  these  independent  variables, 
combinational  measure  wac  the  best  univariate  predictor  of 
performance,  explaining  6  3%  of  Vie  performance  variance. 

This  suggests  that  the  method  used  to  r  ystematically  develop 
ship  combinations  has  a  strong  influence  on  performance  scores. 

Next  in  importance  in  score  prediction  were  processor 
complexity,  and  pre-test  score.  '  >e  of  additional  variables 
increased  the  percentage  of  variance  explained  by  le^  than  1%. 


Table  25  shows  a  regression  ta*-"..  for  the  independent 
variables.  Combinational  scores  correlate  very  hig  '.y  with  the 


Table  22 


Step-Wise  Multiple  Linear  Regressions 
for 

_ Seven  independent  Variables _ 


Number  of 


Independent 

Variables 

Independent  Variables 
Explaining  Largest  Variance 

Variance 

Explained 

1 

Pre-test  Score 

46% 

2 

Pre-test  Score,  Processor 
Complexity 

57% 

3 

Pre-test  Score,  Processor 
Complexity,  Participant 
Category 

60% 

4 

Pre-test  Score,  Processor 
Complexity,  Participant 
Category,  Problem  Complexity 

62% 

5 

Pre-test  Score,  Processor 

65% 

Complexity,  Partic'oant 
Category,  Problem  Complexity, 
Years  Experience 


6  -  /  Less  than  1%  increase  in  the 

amount  of  variance  explained 


Table  23 


Correlation  and  Step-Wise  Regression  with  Five  Independent  Variables 


Correlation*  Regression 

Independent  Standard 


Variable  Coefficient 

Coefficient 

Deviation 

t-Value 

Participant  Categ. 

.437 

.200 

.043 

4. 

663** 

Pre-test  Score 

.674 

.475 

.049 

9. 

694** 

Experience 

-  .206 

-.022 

.006 

-3. 

604** 

Problem  Complexity 

-.156 

-.082 

.026 

-3. 

175*** 

Processor 

-.335 

-.009 

.001 

-7. 

367** 

Complexity 
Intercept  =  .216 

Multiple  Correlation  Coefficient  =  .804 
Explained  Variance  =  .647 
Standard  Error  or  Estimate  =  .230 

Analysis  of  Variance  Explained 

Source  Sum  Sq  DF  Mean  Sq  F-Ratio 


REG  16.220  5  3.24403  61.511  ** 

RES  Li .  oUO  168  .05274 


*Correlation  with  Dependent  Variable:  Sco''c  on  Each  Experiment 
Problem 


** 


1(173)  = 

1(173)  = 

1  (173)  - 

1(173)  = 

F(5,168)= 


4 . 663 ,  p  ^  , 001 
9.694,  £  ^  .001 
-7.367,  p  ^  .001 
-3.604,  p  ^  .001 
61 .51 1  ,  p  ^  .001 


***  1  (173)  =  -  3.175,p^.01 
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Tab le  24 


Step-Wise  Multiple  Linear  Regressions 

for 

Ten  Independent  Variables 


Number  of 


Independent 

Variables 

Independent  Variables 
Explaining  Largest  Variance 

Variance 

Explained 

1 

Combinational  Measure  (CM) 

63% 

2 

CM,  Processor  Complexity 

72% 

3 

CM,  Processor  Complexity, 
Pre-test  Score 

80% 

4-10 

Less  '•ban  !%  increase  in 
amount  of  variance  explained 

Table  25 


Correlation  and  Step-Wise  Linear  Regression  with  Three  Independent  Variables 


Correlation 

Independent 

Variable 

Coefficient 

Regression 

Standard 

C  oefP  clent  De v  i  at  i  on 

t-Value 

Combinat  onal 

Measure 

.796 

.7  58 

.056 

13.584* 

Processor  Complexity 

-.335 

-.008 

-.001 

-  S .821  * 

Pre-test  Score 

.674 

.304 

.038 

7.921* 

Intercept  -  .142 

Multiple  Correlation  Coefficient  =  .891 
Explained  Variance  =  .795 
Standard  Error  of  Estimate  =  .174 


Analysis  of  Variance  Table 


Source 

Sum  Sq 

pr- 

Mean  Sq 

F-Ratio 

REG 

19.927 

3 

C . '  4246 

219.140" 

RES 

5.153 

170 

0.63031 

(173) 

13.584,  p  < 

.001 

(173) 

-  8.821 ,  p  - ' 

.001 

(173)  v  = 

7.921 ,  p  - 

.001 

(3,  170) 

219. 140,  p  < 

.001 

dependent  variable  test  score  (.796).  In  this  rcc,>"ession  the 
coefficients  for  combinational  score  were  f<  jnd  to  be  highly 
significant  (t_  (173)  =  13.664,  p  .001),  as  were  those  for 
pre-test  scores  (t_  (173)  --  7.931,  p  ^.001)  and  processor 

complexity  (t_(173)-  '  ,p  .001)-  The  regression  equation 

was  also  found  to  be  significant  (F  (3,170  )  =219.140  ,  p  ^.00l). 

DISCUSSION  AND  CONCLUSIONS 

Cased  on  the  data  analyses  reported  in  the  previous 
section,  the  fo"owinc  conclusions  are  offered: 

1  .  Specification  of  problem  solutions  via  example 

solutu  s  results  in  a  low  rate  of  errors  of  com- 
missio  -  as  evidenced  by  the  similar  performance 
results  obtained  from  Performance  Measurement 
1  and  Performance  Measurement  2.  These  re¬ 
sults  do  not  suggest  that  input  errors  were  not 
made  in  the  experiments.  Indeed,  input  errors 
were  made,  but  most  were  detect'  '  and  corrected 
as  work  on  the  problem  continued. 

Additional  work  is  planned  to  compare  subject  error 
rates  using  example  solutions,  as  in  a  us  study,  to 
other  ways  of  specifying  problem  solutions. 

2 .  Programmers  as  a  group  performed  significantly 
better  than  d  *  bookkeepers/accounta:  its  on  problems 
studied  in  this  effort.  I  his  is  evident  both  from 
the  ANOVA  results  and  from  a  simple  inspection 
of  .  ie  raw  data.  The  experiment  problem  format 
was  designed  to  be  neutral;  i.e.,  fi  .  oring  neither 
programmers  nor  bookkeepers/accountants .  Both 

participant  categories  are  familiar  with 
accurate  and  detailed  work;  however,  bookkeepers/ 
accountants  may  be  used  to  mo>-  :  ixed  or  structured 
problems  whereas  programmers  may  ue  more  at 
ease  with  a  variety  of  problems.  Thus,  the 
novelty  of  the  problem  may  have  uvored  program¬ 
mers  over  bookkeepers/,  countants.  Further, 
programmers  may  be  more  comfortable  with 
computer  displays;  however,  bookkeepers/ accountants 
may  be  more  familiar  (and  may  t;  icrefore  enter 
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data  faster)  with  a  key  pad.  (A  key  pad  has  only 
numerals  a>  ranged  in  the  calculator  " matrix " 
format.)  Programmers  appear  to  be  more  familiar 
with  'ypewritei — like  key  arrangements ,  often  using 
two  fingers  to  enter  data.  The  extent  to  which 
these  aspects  of  the  situation  have  affected  the 
results  is  not  known.  However,  since  the  combinational 
measure  m  .  univariate  regre  sion  explained  more 
variance  than  did  participant  category  (i.e., 
programmers,  bookkeepers/accountants),  the  use  of 
systematic  mother  ...  of  generating  ship  combinations 
is  more  important  for  performance  than  participant 
category;  and  a  high  percentage  of  programmers 
used  that  systematic  method.  Further,  it  is  likely 
that  prog^  Timers  would  be  more  familiar  with 
efficient  combination-gene  mting  analogs  such  as  the 
binary  number  system.  Adoitional  work  is  planned 
using  Lodback  aids  which  present  combinations  in 
usable  forms  to  participants  to  determine  if  per¬ 
formance  in  both  participant  categories  can  be 
improved. 

3.  It  is  clear  that  problem  complexity  and  processor 

complexity  significantly  affect  performance.  In 

. .  ition,  the  two-way  multiple  comparison  test 

results  (SNK)  reveal  that  the  level  of  processor 

complexity  was  the  dominant  factor  affecting 

performance  of  both  participant  categories. 

Programmers  working  with  Processor  Level  B  , 

which  provides  very  little  automatic  processing3 

and  thus  little  generalization  of  t!  innut  data, 

experienced  a  s.gnificant  reduction  of  performance. 

This  reduction  of  performance  occurred  for  all 

levels  of  problem  complexity.  Programmers 

performed  significantly  better  with  Processor  Level 

B  ,  which  provides  considerable  generalization 

of  the  input  data.  Also,  programmers  performed 

significantly  better  with  Processor  '  vel  B  , 

provided  that  the  problems  were  not  difficult, 

i.e.,  provided  that  Problem  Levels  A  and  A 

1  2 

were  used.  In  contrast,  the  bonkkeepers/accountants 
performed  significantly  better  with  Processor 

Level  B  on  Problem  Levels  A  nd  Ar  only. 

I  s  3 
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Bookkeepers/accountants  failed  to  show  a  sig¬ 
nificant  improvemenl  in  performance  with  Processor 

Level  '3  over  Processor  Level  B  .  A  low  rate 
2  3 

of  error;  >f  commission  was  observed  from  both 

participant  categories  using  example  solutions.  This 
may  be  a  property  of  specifying  solutions  witn 
examples  instead  of  a  more  general  specification. 
However,  the  performance  measures  used  (PM  1 
and  PM  2)  combine  the  effects  both  of  the  selection 
of  ship  comoinations  and  the  development  of  a 
range  of  transiting  and  stationing  times  in  combina¬ 
tion  to  form  the  SSL.  It  would  be  expected  that 
the  use  of  ocreased  processor  generalization  might 
improve  the  continuous  or  parametric  portions  of 
the  specification  (i.e.,  the  correct  range  of  transiting 
and  stationing  times),  but  would  not  affect  the 
frequency  with  which  f  ctors  are  incorrectly 
omitted  (errors  of  omission)  or  incorrectly 
included  (errors  of  commission).  It  is 
further  expect  mat,  as  stated  previously,  the 
use  of  feedback  aids  structured  to  encourage  systematic 
generation  of  (actor  combinations  (i.e.,  ship 
combination.)  might  aid  in  reducing  errors  of 
omission.  Additional  analysis  and  experiments 
are  planned  to  further  investigate  those  areas. 

I  tie  first  regression  analysis  (Analysis  5)  showed 
that  pre-test  score  was  the  single  most 
important  factor  of  the.  sevm  factors  analyzed  in 
predicting  perform  ance  scores.  Pre-test  score, 
obtained  on  a  common  problem  and  problem  complexity 
cell,  may  measure  inherent  participant  ability 
and  is  a  strong  factor  in  predicting  performance. 

It  is  interesting  to  note  "  at  participant  experience, 
measured  in  years  of  professional  maining  (and/or 
work), was  not  a  strong  factor  in  prediction  the 
performance  score  variance.  This  result 
suggests  that  the  skills  required  to  perform  well 
on  the  e>  -riment  problems  are  not  skills 
naturally  developed  by  years  of  training  and 
work  experience.  But,  it  may  also  indicate 
that  skills  taught  some  time  ago,  plus  skills 
learned  over  years  of  experience,  are  of  equal 
value  to  skills  fumed  recently. 
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6. 


Participant  procedural  strategies  measured  by 
P  and  Po  explained  a  substantial  portion  of 
the  score  variance  in  a  univariate  regression. 
However,  since  P^  was  positively  and  P^  negatively 
correlated  with  performance,  it  appears  that 
most  participants  tended  to  generate  and  enter 
multiple  input  examples  before  checking  the  effect 
of  these  inputs,  i.e.,  successful  participants 
developed  a  lan  and  correctly  interpreted  the 
effects  of  entering  multiple  example  solutions . 

7.  Since  the  combinational  measure  explained  63% 

of  the  score  variance  and  together  with  processor 
complexity  and  pre-test  score  explained  80%  of 
the  score  variance,  Lhe  use  of  a  systematic 
method  for  identifying  and  generating  combinations 
of  ships  must  be  a  key  skill  in  solving  the  ex¬ 
periment  problems.  Indeed,  the  r ombinational 
strategy  measure  has  a  higher  correlation  with 
the  performance  score  than  does  participant 
category.  Apparently,  therefore,  a  systematic 
strategy  was  used  both  by  successful  programmers 
and  bookkeepers/accountants  to  achieve  high 
scores . 
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APPENDIX  A 


TEST  PROBLEM  #  93 


Now  suppose  the  mission  has  been  modified.  You  are 
asked  to  develop  the  example  solutions  required  to  specify  the  ship 
selection  logic  for  the  modified  mission.  The  selection  criteria  for 
the  task  force  has  been  modified  to: 

1 .  The  ships  needed  for  the  task  force  are: 

•  2  Attack  Aircraft  Carriers  (CVA  or  CVAN),  and 

•  2  Submarines  (SS) 

2.  The  present  position  of  the  ships  and  their 
speed  must  permit  the  ships  to  arrive  at  the 
assembly  point  within  5  days. 

3.  Supplies  on  board  each  ship,  including  fuel, 
must  permit  stationing  without  resupply  for 
a  10  day  period. 

This  task  force  criteria  specifies  3  combinations  of  ship 
types  as  follows: 

•  2  CVA  and  2  SS 
or 

•  2  CVAN  and  2  SS 
or 

•  1  CVA  and  1  CVAN  and  2  SS 

You  must  develop  example  solutions  for  each  ship  combination  using  ships 
with  suitable  transit  and  stationing  times. 
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Harry's  example  input  list  has  been  expanded  and  input 
to  the  computer.  Please  use  the  expanded  example  input  list  (to 
identify  ships  by  their  item  number)  to  develop  example  solutions, 
enter  the  solutions  into  the  computer,  and  test  the  computer's  gen¬ 
eralization  of  your  examples  by  requesting  additional  solutions  for 
the  computer.  If  a  computer  solution  is  in  error,  develop  and 
enter  additional  solutions  using  the  aids  the  computer  automatically 
provides.  Continue  this  process  (inputting  example  solutions  and 
testing  the  computer)  until  you  are  satisfied  that  the  computer  has 
properly  generalized  your  solution  examples,  i.e.,  it  uses  your 
examples  to  provide  its  own  correct  solutions. 


PLEASE  START  NOW. 


WHEN  YOU  HAVE  FINISHED: 


When  you  believe  the  computer  SSL  is  correct,  give  it 

this  test: 

Would  the  SSL  select  the  following  ships  for  the 
task  force? 


CVA 

CVAN 

SS 

SS 


T ransit  T  ime 
3  days 

2  days 

3  days 
5  days 


Yes 


No 

_ 


Stationing  Time 
17  days 
75  days 
1 1  days 
25  days 
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TEST  PROBLEM  #31 


Now  suppose  the  mission  has  been  modified.  You  are 
asked  to  develop  me  example  solutions  required  to  specify  the  ship 
selection  logic  for  the  modified  mission.  The  selection  criteria  for 
the  task  force  has  been  modified  to: 

1  .  The  ships  needed  for  the  task  force  are: 

•  2  Attack  Aircraft  Carriers  (CVA)  or  1 
Nuclear  Attack  Aircraft  Carrier  (CVAN),  and 

•  (2  Submarines  (SS  or  SSN)  and  3 
Destroyers  (DD))  or  (3  Submarines 
(SS  o'-  SSN)  and  2  Destroyers  (DD)) 
and 

•  1  Oiler  (AO). 

2.  The  present  position  of  the  ships  and  their 
speed  must  permit  the  ships  to  arrive  at  the 
assembly  point  within  5  days. 

3.  Supplies  on  board  each  ship,  including  fuel, 
must  permit  stationing  without  resupply  for 
a  10  day  period. 

Harry's  example  input  list  has  been  expanded  and  input 


to  the  computer.  Please  use  the  expanded  example  input  list  (to 


identify  ships  by  their  item  number)  to  develop  example  solutions, 
enter  the  solutions  into  the  computer,  and  test  the  computer’s  gen¬ 
eralization  of  your  examples  by  requesting  additional  solutions  for 
the  computer.  If  a  computer  solution  is  in  error,  develop  and 
enter  additional  solutions  using  the  aids  the  computer  automatically 
provides.  Continue  this  process  (inputting  example  solutions  and 
testing  the  computer)  until  you  are  satisfied  that  the  computer  has 
properly  generalized  your  solution  examples,  i.e.,  it  uses  your 
examples  to  provide  its  own  correct  solutions. 

PLEASE  START  NOW. 
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TEST  PROBLEM  #15 


Now  suppose  the  mission  has  been  modified.  You  are 
asked  to  develop  example  solutions  for  the  modified  mission.  The 
selection  criteria  for  the  task  force  has  been  modified  to: 

1 .  The  ships  needed  for  the  task  force  are: 

•  1  Attack  Aircraft  Carrier  (CVA  or  CVAN),  and 

•  2  Submarines  (SS  or  SSN),  and 

•  4  Destroyers  (DD),  and 

•  1  Oiler  (AO) 

2.  The  present  position  of  the  ships  and  their 

speed  must  permit  the  ships  to  arrive  at 
the  assembly  point  within  5  days. 

3.  Supplies  on  board  each  ship,  including  fuel, 
must  permit  stationing  without  resupply  for 
a  10  day  period. 

Harry's  example  input  list  has  been  expanded  and  input 
to  the  computer.  Please  use  the  expanded  example  input  list  (to 
identify  ships  by  their  item  number)  to  develop  example  solutions, 
enter  the  solutions  into  the  computer,  and  test  the  computer's  gen¬ 
eralization  of  your  examples  by  requesting  additional  solutions  for  the 
computer.  If  a  computer  solution  is  in  error,  develop  and  enter  ad¬ 
ditional  solutions  using  the  aids  the  computer  automatically  provides. 
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Continue  this  process  (inputting  example  solutions  and  testing  the 
computer)  until  you  are  satisfied  that  the  computer  has  properly 
generalized  your  solution  examples,  i.e.,  it  uses  your  examples 
to  provide  its  own  correct  solutions. 

PLEASE  START  NOW. 
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TEST  PROBLEM  #52 


Now  suppose  the  mission  has  been  modified.  You  are 
asked  to  develop  example  solutions  for  the  modified  mission.  The 
selection  criteria  for  the  task  force  has  been  modified  to: 

1  .  The  ships  needed  for  the  task  force  are: 

•  1  Attack  Aircraft  Carrier  with  Nuclear  Propul¬ 
sion  (CVAN),  and 

•  2  Guided  Missile  Cruisers  (CG  or  CGN),  and 

•  2  Submarines  (SS  or  SSN),  and 

•  3  Destroyers  (DD),  and 

•  2  Oilers  (AO) 

2.  The  present  position  of  the  ships  and  their  speed 
must  permit  the  ships  to  arrive  at  the  assembly 
point  within  5  days. 

3.  Supplies  on  board  each  ship,  including  fuel, 
must  permit  stationing  without  resupply  for  a 
10  day  period. 

Harry's  example  input  list  has  been  expanded  and  input 
to  the  computer.  Please  use  the  expanded  example  input  list  (to 
identify  ships  by  their  item  number)  to  develop  example  solutions, 
enter  the  solutions  into  the  computer,  and  test  the  computer's  gen¬ 
eralization  of  your  examples  by  requesting  additional  solutions  for 
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the  computer.  If  a  computer  solution  is  in  error,  develop  and 
enter  additional  solutions  using  the  aids  the  computer  automatically 
provides.  Continue  this  process  (inputting  example  solutions  and 
testing  the  computer)  until  you  are  satisfied  that  the  computer  has 
properly  generalized  your  solution  examples,  i.e.,  it  uses  your 
examples  to  provide  its  own  correct  solutions. 

PLEASE  START  NOW. 
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APPENDIX  B 
participant  CRITERIA 


/ 


Programmer  Selection  Criteria 


Programmers  selected  to  participate  in  the  experiment 
should  have  the  following  experience: 

1  .  Coding  and  Testing  Experience: 

Participants  shall  have  experience  in  at  least  two 
higher  level  languages  from  among  the  following  list: 

a.  BASIC 

b.  FORTRAN 

c.  COBOL 

d.  PL-1 

e.  ALGOL 

f.  PASCAL 


Participants  shall  have  been  responsible  for  the  entire 
process  of  coding  and  testing  the  programs  and  shall 
have  produced  at  least  10,000  lines  of  program  code 
(100  programs  of  at  least  100  lines  each). 

2.  Computer  Systems: 


Participants  shall  have  programmed  with  two  different 
computer  systems,  i.e.,  OS  and  DOS,  or  equipment 
from  different  manufacturers,  such  as  Honeywell,  IBM, 
DEC ,  etc . 

3.  Timesharing 

Participants  shall  have  at  least  160  hours  of  on-line 
terminal  usage  on  a  timesharing  system. 

4.  Application  Systems: 

Participants  shall  have  coded  at  least  two  systems  of 
programs  of  at  least  ten  programs  each. 
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5. 


JCL: 


Participants  shall  have  been  responsible  for  at  least 
twenty  programs . 

6.  Participants  shall  have  used  "modular"  or  "structured" 
programming  techniques . 

7.  Each  participant  must  be  over  18  years  of  age. 

8.  English  must  be  the  participants'  native  language. 
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Bookkeeper/Accountant  Selection  Criteria 


Bookkeepers  and  accountants  selected  to  participate  in 
the  experiment  shall  have  the  following  experience  and  qualifica¬ 
tions: 

1.  Participants  must  be  over  18  years  of  age. 

2.  English  must  be  the  participants’  native  language. 

3 .  Participants  should  never  have  been  a  professional 
computer  programmer. 

4.  Participants  should  have  at  least  2  years  of  book¬ 
keeping/accounting  schooling  AND  2  years  of  book¬ 
keeping/accounting  work  experience, 

OR 

5.  Participants  should  have  at  least  4  years  of  book¬ 
keeping/accounting  work  experience, 

OR 

6.  Participants  should  have  at  least  4  years  of  book¬ 
keeping/accounting  schooling. 
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appendix  c 

newspaper  advertisement 


NEWSPAPER  ADVERTISEMENT 


Accountants/Bookkeepers 

A  firm  in  Vienna  needs  the  help  of  accountants  and  bookkeepers 
in  evaluating  a  computer  system  for  use  by  non  programmers. 
4-5  hour  evaluation.  4  years  bookkeeping/accounting  schooling 
or  experience  needed.  $7.50  per  hour.  938-1600. 
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APPENDIX  D 

PARTICIPANT  BIOGRAPHICAL  FORM 


C-36 

FOR  OFFICIAL  USE  ONLY 


I - 1 

I  Participant  No. _  I 


*  Date: 
I 


PARTICIPANT  BIOGRAPHICAL  FORM 


Name: 

Age: 

Education: 


_  Sex:  _ 

High  School  Gradu.  'e:  Yes  _  No  _ 

_  years  of  higher  education  Degree: 

Major:  _ _  Minor: 


Bookkeeping  and  Accounting  Experience 

Please  check  all  areas  in  which  you  have  worked  and  enter  the  number 
of  months  of  full-time  experience  in  that  area  (2  years  part  time  =  1  year 
full  time). 

_ Accounts  Payable/Cash  Disbursements  _ 

_ Accounts  Receivuble/Cash  Receipts 

_ Trial  Balance  Preparation 

_ Preparation  of  f’oriodic  Reports  (Income  Statement, 

Balance  Sheets,  etc.)  _ 

_ Accounting  for  Inventory 

_ Payroll 

_ T  axes 

Circle  the  number  of  years  in  which  you  have  worked  a^  a  full-time 
bookkeeper  or  accountant:  1  2  34  567  89  10  10 

Please  make  any  additional  comments  you  believe  best  describes  your 
professional  expertise. 


/ 
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Higher  Level  Language  Programming  Experience 


Please  circle  all  languages  in  which  you  have  programmed  and  check 
your  best  estimate  of  the  number  of  programs  coded  in  that  language. 

TYPE  1-10  11-20  21-30  31-50  51-100  100* 

BASIC 

FORTRAN 

COBOL 

PL-1 

ALGOL 

PASCAL 


Circle  your  best  estimate  of  the  total  number  of  programs  you  have 
coded  (any  language,  any  computer). 

0  1-100  101-200  201-300  301-500  500* 


Data  Prot-wssing  Experience 

Check  all  areas  in  which  you  have  worked  and  enter  the  number  of 
months  of  full-time  experience  in  that  area  (2  years  part  time  =  1  year 
full  time). 

_ Data  Entry  _  _ System  Analysis 

Production  Control  Data  Base  Administration 


_Operations  _  _ Data  Communications 

Applications  Programming _  _ Other(s) 

System  Programming 


Circle  the  number  of  years  in  which  you  have  worked  as  a  full-time 
data  processing  professional:  1  234567891010+ 
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Circle  your  best  estimate  of  the  number  of  programs  for  which  you 
have  coded  the  Job  Control  Language  (JCL)  necessary  for  testing  or  produc¬ 
tion  runs. 

1-10  11-20  21-30  31-50  51-100  100+ 


Please  list  up  to  5  computer/operating  systems  on  which  you  have 
worked  which  you  believe  best  represents  your  experience.  Examples: 
IBM  370/155  OS,  PDP-11  RT-1  1  . 
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APPENDIX  E 

SAMPLE  RELEASE  FORM 


CONTRACT  TO  ACT  AS  A  RESEARCH  SUBJECT 
FOR  THE  DEPARTMENT  OF  THE  NAVY 


NAME  OF  PARTICIPANT _  DATE: _ 

(Please  Print) 

ADDRESS: _ TELEPHONE: _ 

1.  I  authorize  Mr.  Edward  M.  Connelly  who  is  the  Principal  Investigator  for  this 
project  or  his  representatives,  Mr.  Bob  Comeau  or  Mrs.  Joanne  Connelly 

to  collect  and  analyze  my  examples  of  solutions  to  test  problems. 

2.  This  project  has  been  explained  to  me  by  _ . 

(Print) 

It  has  been  pointed  out  to  me  that  my  solution  examples  will  not  be  made  known 
to  any  individual  ,  other  than  appropriate  members  of  the  research  team,  or 
for  any  purpose  other  than  the  data  analysis  to  be  performed  by  the  research 
team.  No  information  concerning  my  performance  on  the  experimental  task 
will  be  disclosed  to  anyone  other  than  the  research  team  without  my  written 
permission. 

S.  I  understand  that  there  are  no  special  risks  of  any  kind  associated  with  my 
participation  in  this  study. 

4.  I  understand  that  the  project  may  further  the  understanding  of  how  various 
programming  aids  can  assist  a  programmer  or  other  person  to  specify  a 
computer  program . 

5.  I  understand  that  Mr.  Edward  M.  Connelly,  Mr.  Bob  Comeau,  Mrs. 

Joanne  Connelly  will  answer  any  questions  I  may  have  about  the  project. 

6.  I  understand  that  by  signing  this  form,  1  have  waived  none  of  my  legal  rights 
that  may  be  associated  with  liability  for  negligence  on  the  part  of  Performance 
Measurement  Associates,  Inc. 

7 .  I  state  that  I  am  1 8  years  of  age  or  older . 


Subject: 


(Signature) 


Witness: 


(Signature) 


Edward  M.  Connelly 
Principal  Investigator 
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SHIP  LIST 


Ship  ID  Number 

Ship  Type 

Transit  Time 

Station  T ime 

1 

CVAN 

2 

11 

2 

CVAN 

5 

11 

3 

CVAN 

2 

19 

4 

CVAN 

2 

112 

5 

CVAN 

4 

10 

6 

CVAN 

4 

176 

7 

CVAN 

1 

43 

a 

CVAN 

3 

72 

9 

CVAN 

3 

200 

10 

CVAN 

2 

148 

1 1 

CVAN 

3 

168 

12 

CVAN 

3 

10 

13 

CVAN 

3 

69 

14 

CVAN 

1 

158 

15 

CVAN 

2 

193 

16 

CVAN 

4 

69 

17 

CVAN 

3 

145 

18 

CVAN 

3 

149 

19 

CVAN 

5 

179 

20 

CVAN 

3 

200 

21 

CVA 

2 

20 

22 

CVA 

3 

15 

23 

CVA 

2 

16 

24 

CVA 

4 

17 

25 

CVA 

3 

17 

26 

CVA 

3 

13 

27 

CVA 

5 

17 

28 

CVA 

2 

16 

29 

CVA 

4 

18 

30 

CVA 

2 

16 

31 

CVA 

1 

14 

32 

CVA 

2 

10 

33 

CVA 

4 

18 

34 

CVA 

5 

12 

35 

CVA 

2 

18 

36 

CVA 

4 

19 

37 

CVA 

1 

19 
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Ship  IQ  Number  Ship  Type  Transit  Time  Station  Time 


38 

CVA 

2 

11 

39 

CVA 

2 

10 

40 

CVA 

4 

20 

41 

CA 

4 

29 

42 

CA 

4 

30 

43 

CA 

4 

29 

44 

CA 

4 

23 

45 

CA 

3 

24 

46 

CA 

5 

14 

47 

CA 

4 

22 

48 

CA 

4 

10 

49 

CA 

2 

30 

50 

CA 

4 

26 

51 

CA 

3 

29 

52 

CA 

2 

28 

53 

CA 

2 

29 

54 

CA 

1 

19 

55 

CA 

1 

19 

56 

CA 

4 

10 

57 

CA 

3 

23 

58 

CA 

3 

28 

59 

CA 

4 

20 

60 

CA 

5 

26 

61 

CGN 

4 

289 

62 

CGN 

1 

155 

63 

CGN 

4 

46 

64 

CGN 

4 

228 

65 

CGN 

2 

147 

66 

CGN 

5 

215 

67 

CGN 

2 

300 

68 

CGN 

3 

10 

69 

CGN 

2 

17 

70 

CGN 

4 

148 

71 

CGN 

3 

243 

72 

CGN 

4 

290 

73 

CGN 

3 

150 

74 

CGN 

2 

191 

75 

CGN 

2 

10 

76 

CGN 

2 

300 

77 

CGN 

3 

237 
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Ship  ID  Number  Ship  Type  Transit  Time  Station  Time 


78 

CGN 

3 

154 

79 

CGN 

1 

194 

80 

CGN 

5 

102 

81 

CG 

3 

20 

82 

CG 

3 

14 

83 

CG 

2 

14 

84 

CG 

2 

16 

85 

CG 

2 

28 

86 

CG 

5 

14 

87 

CG 

2 

18 

88 

CG 

3 

10 

89 

CG 

2 

30 

90 

CG 

4 

30 

91 

CG 

3 

25 

92 

CG 

2 

21 

93 

CG 

5 

18 

94 

CG 

4 

10 

95 

CG 

2 

27 

96 

CG 

2 

14 

97 

CG 

2 

16 

98 

CG 

1 

27 

J9 

CG 

1 

14 

100 

CG 

4 

29 

101 

DD 

3 

10 

102 

DD 

5 

26 

103 

OD 

2 

19 

104 

DD 

2 

24 

105 

DD 

3 

26 

106 

DD 

4 

27 

107 

DD 

1 

21 

108 

DD 

2 

24 

109 

DD 

2 

11 

110 

DD 

2 

29 

111 

DD 

4 

23 

112 

DD 

2 

30 

113 

DD 

3 

21 

114 

DD 

4 

17 

115 

DD 

3 

30 

116 

DD 

4 

10 

117 

DD 

1 

12 
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Ship  ID  Number 


Ship  Type 


Transit  Time 


Station  Time 


118 

DD 

5 

19 

119 

DD 

4 

19 

120 

DD 

3 

13 

121 

DD 

4 

19 

122 

DD 

3 

25 

123 

DD 

5 

23 

124 

DD 

4 

29 

125 

DD 

4 

16 

126 

DD 

4 

29 

127 

DD 

4 

30 

128 

DD 

4 

18 

129 

DD 

2 

29 

130 

DD 

5 

23 

131 

DD 

1 

15 

132 

DD 

2 

30 

133 

DD 

2 

10 

134 

DD 

2 

22 

135 

DD 

3 

10 

136 

DD 

3 

11 

137 

DD 

1 

28 

138 

DD 

4 

19 

139 

DD 

4 

19 

140 

DD 

4 

16 

141 

SSN 

2 

226 

142 

SSN 

3 

192 

143 

SSN 

2 

17 

144 

SSN 

3 

10 

145 

SSN 

4 

152 

146 

SSN 

2 

210 

147 

SSN 

2 

300 

148 

SSN 

3 

111 

149 

SSN 

4 

136 

150 

SSN 

4 

229 

151 

SSN 

3 

181 

152 

SSN 

1 

156 

153 

SSN 

4 

148 

154 

SSN 

3 

300 

155 

SSN 

2 

10 

156 

SSN 

4 

295 

157 

SSN 

5 

47 
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